Bug 1723051 - Amphora loadbalancers fail to create
Summary: Amphora loadbalancers fail to create
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 15.0 (Stein)
Assignee: Nir Magnezi
QA Contact: Bruna Bonguardo
URL:
Whiteboard:
: 1736267 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-22 14:07 UTC by Carlos Goncalves
Modified: 2019-09-26 10:52 UTC (History)
11 users (show)

Fixed In Version: openstack-tripleo-common-10.8.1-0.20190710191707.b6a2d65.el8ost puppet-octavia-14.4.1-0.20190710190429.381f1a3.el8ost openstack-tripleo-heat-templates-10.6.1-0.20190711090428.245f17c.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:23:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 666971 0 'None' MERGED Ensure that OctaviaServerCertsKeyPassphrase is 32-byte long 2020-12-08 12:43:38 UTC
OpenStack gerrit 666987 0 'None' MERGED Adds constraint: OctaviaServerCertsKeyPassphrase must be 32 chars long 2020-12-08 12:44:07 UTC
OpenStack gerrit 666989 0 'None' MERGED Configure 32 chars length server_certs_key_passphrase for Octavia 2020-12-08 12:43:39 UTC
OpenStack gerrit 669653 0 'None' MERGED Configure 32 chars length server_certs_key_passphrase for Octavia 2020-12-08 12:43:39 UTC
OpenStack gerrit 669657 0 'None' MERGED Ensure that OctaviaServerCertsKeyPassphrase is 32-byte long 2020-12-08 12:43:38 UTC
OpenStack gerrit 669667 0 'None' MERGED Adds constraint: OctaviaServerCertsKeyPassphrase must be 32 chars long 2020-12-08 12:43:39 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:23:51 UTC

Description Carlos Goncalves 2019-06-22 14:07:06 UTC
Description of problem:

Amphora load balancers fail to create.
Worker log show "Fernet key must be 32 url-safe base64-encoded bytes.

Version-Release number of selected component (if applicable):
OSP 15 RHOS_TRUNK-15.0-RHEL-8-20190619.n.1

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP 15 with Octavia enabled
2. Create loadbalancer


Actual results:

[...]
2019-06-21 03:13:59.870 18 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' (2b6a2a25-396a-42bf-9f91-0e9fb1014deb) transitioned into state 'FAILURE' from state 'RUNNING'
12 predecessors (most recent first):
  Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': '7a8d9ce6-e033-4726-aac8-534b80843f1e'}
  |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
     |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '41e00d9d-ad66-4c46-a49d-e8cb852e0d9f', 'flavor': {}}, 'provides': None}
        |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
           |__Flow 'STANDALONE-octavia-plug-net-subflow'
              |__Atom 'octavia.controller.worker.tasks.network_tasks.GetSubnetFromVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0da0>}, 'provides': <octavia.network.data_models.Subnet object at 0x7f0ff4098208>}
                 |__Atom 'octavia.controller.worker.tasks.network_tasks.UpdateVIPSecurityGroup' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0da0>}, 'provides': None}
                    |__Atom 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '41e00d9d-ad66-4c46-a49d-e8cb852e0d9f', 'vip': <octavia.common.data_models.Vip object at 0x7f0ff4c033c8>}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0da0>}
                       |__Atom 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0ef0>}, 'provides': <octavia.common.data_models.Vip object at 0x7f0ff4c033c8>}
                          |__Atom 'reload-lb-before-allocate-vip' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '41e00d9d-ad66-4c46-a49d-e8cb852e0d9f'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0ef0>}
                             |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '41e00d9d-ad66-4c46-a49d-e8cb852e0d9f'}, 'provides': None}
                                |__Flow 'octavia-create-loadbalancer-flow': ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker     result = task.execute(**arguments)
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/controller/worker/tasks/cert_task.py", line 50, in execute
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker     fer = fernet.Fernet(key)
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker   File "/usr/lib64/python3.6/site-packages/cryptography/fernet.py", line 37, in __init__
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker     "Fernet key must be 32 url-safe base64-encoded bytes."
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker 
2019-06-21 03:13:59.880 18 DEBUG octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' (2b6a2a25-396a-42bf-9f91-0e9fb1014deb) transitioned into state 'REVERTING' from state 'FAILURE' _task_receiver /usr/lib/python3.6/site-packages/taskflow/listeners/logging.py:194
2019-06-21 03:13:59.882 18 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' (2b6a2a25-396a-42bf-9f91-0e9fb1014deb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
2019-06-21 03:13:59.884 18 DEBUG octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' (2440e1c3-e310-44cd-8eb9-64a9cd20b85b) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python3.6/site-packages/taskflow/listeners/logging.py:194
2019-06-21 03:13:59.884 18 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting create amphora in DB for amp id 7a8d9ce6-e033-4726-aac8-534b80843f1e 
2019-06-21 03:13:59.901 18 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' (2440e1c3-e310-44cd-8eb9-64a9cd20b85b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
[...]

Comment 3 Carlos Goncalves 2019-06-22 15:17:06 UTC
The Fernet key in the faulty deployment is not 32 bytes. The key was automatically generated by TripleO and is "InD5Bw4Qv37uG3W1Jx7ifkA5i" (25 bytes).

$ grep server_certs_key_passphrase var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf 
server_certs_key_passphrase=InD5Bw4Qv37uG3W1Jx7ifkA5i

A simple reproducer is:

import base64
import six
from cryptography import fernet

# 32-byte key (good)
# key = "insecure-key-do-not-use-this-key"

# not 32-byte key (bad)
key = "InD5Bw4Qv37uG3W1Jx7ifkA5i"

if six.PY3 and isinstance(key, six.string_types):
    key = key.encode('utf-8')
key_b64 = base64.urlsafe_b64encode(key)
fernet.Fernet(key_b64

Comment 4 Carlos Goncalves 2019-06-22 15:44:18 UTC
Two possible fix options here:

1. Ensure the key is padded to 32 bytes if shorter but also not trimmed if longer. Alternatively, invalidate if not exactly 32-byte long.
2. Ensure TripleO generates a key with an acceptable size. If key provided by the operator, set constraints in order to validate it at deployment time (rather than at Octavia service start up time, i.e. fail early)

Either way, we might have to explicitly document what is the key size expectations and validate its content at service startup.

Comment 17 Carlos Goncalves 2019-08-01 16:51:24 UTC
*** Bug 1736267 has been marked as a duplicate of this bug. ***

Comment 22 errata-xmlrpc 2019-09-21 11:23:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.