Description of problem: Amphora load balancers fail to create. Worker log show "Fernet key must be 32 url-safe base64-encoded bytes. Version-Release number of selected component (if applicable): OSP 15 RHOS_TRUNK-15.0-RHEL-8-20190619.n.1 How reproducible: 100% Steps to Reproduce: 1. Deploy OSP 15 with Octavia enabled 2. Create loadbalancer Actual results: [...] 2019-06-21 03:13:59.870 18 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' (2b6a2a25-396a-42bf-9f91-0e9fb1014deb) transitioned into state 'FAILURE' from state 'RUNNING' 12 predecessors (most recent first): Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': '7a8d9ce6-e033-4726-aac8-534b80843f1e'} |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow' |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '41e00d9d-ad66-4c46-a49d-e8cb852e0d9f', 'flavor': {}}, 'provides': None} |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow' |__Flow 'STANDALONE-octavia-plug-net-subflow' |__Atom 'octavia.controller.worker.tasks.network_tasks.GetSubnetFromVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0da0>}, 'provides': <octavia.network.data_models.Subnet object at 0x7f0ff4098208>} |__Atom 'octavia.controller.worker.tasks.network_tasks.UpdateVIPSecurityGroup' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0da0>}, 'provides': None} |__Atom 'octavia.controller.worker.tasks.database_tasks.UpdateVIPAfterAllocation' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '41e00d9d-ad66-4c46-a49d-e8cb852e0d9f', 'vip': <octavia.common.data_models.Vip object at 0x7f0ff4c033c8>}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0da0>} |__Atom 'octavia.controller.worker.tasks.network_tasks.AllocateVIP' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0ef0>}, 'provides': <octavia.common.data_models.Vip object at 0x7f0ff4c033c8>} |__Atom 'reload-lb-before-allocate-vip' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '41e00d9d-ad66-4c46-a49d-e8cb852e0d9f'}, 'provides': <octavia.common.data_models.LoadBalancer object at 0x7f0ff40f0ef0>} |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': '41e00d9d-ad66-4c46-a49d-e8cb852e0d9f'}, 'provides': None} |__Flow 'octavia-create-loadbalancer-flow': ValueError: Fernet key must be 32 url-safe base64-encoded bytes. 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last): 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker result = task.execute(**arguments) 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3.6/site-packages/octavia/controller/worker/tasks/cert_task.py", line 50, in execute 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker fer = fernet.Fernet(key) 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker File "/usr/lib64/python3.6/site-packages/cryptography/fernet.py", line 37, in __init__ 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker "Fernet key must be 32 url-safe base64-encoded bytes." 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker ValueError: Fernet key must be 32 url-safe base64-encoded bytes. 2019-06-21 03:13:59.870 18 ERROR octavia.controller.worker.controller_worker 2019-06-21 03:13:59.880 18 DEBUG octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' (2b6a2a25-396a-42bf-9f91-0e9fb1014deb) transitioned into state 'REVERTING' from state 'FAILURE' _task_receiver /usr/lib/python3.6/site-packages/taskflow/listeners/logging.py:194 2019-06-21 03:13:59.882 18 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' (2b6a2a25-396a-42bf-9f91-0e9fb1014deb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' 2019-06-21 03:13:59.884 18 DEBUG octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' (2440e1c3-e310-44cd-8eb9-64a9cd20b85b) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python3.6/site-packages/taskflow/listeners/logging.py:194 2019-06-21 03:13:59.884 18 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting create amphora in DB for amp id 7a8d9ce6-e033-4726-aac8-534b80843f1e 2019-06-21 03:13:59.901 18 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' (2440e1c3-e310-44cd-8eb9-64a9cd20b85b) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' [...]
The Fernet key in the faulty deployment is not 32 bytes. The key was automatically generated by TripleO and is "InD5Bw4Qv37uG3W1Jx7ifkA5i" (25 bytes). $ grep server_certs_key_passphrase var/lib/config-data/puppet-generated/octavia/etc/octavia/octavia.conf server_certs_key_passphrase=InD5Bw4Qv37uG3W1Jx7ifkA5i A simple reproducer is: import base64 import six from cryptography import fernet # 32-byte key (good) # key = "insecure-key-do-not-use-this-key" # not 32-byte key (bad) key = "InD5Bw4Qv37uG3W1Jx7ifkA5i" if six.PY3 and isinstance(key, six.string_types): key = key.encode('utf-8') key_b64 = base64.urlsafe_b64encode(key) fernet.Fernet(key_b64
Two possible fix options here: 1. Ensure the key is padded to 32 bytes if shorter but also not trimmed if longer. Alternatively, invalidate if not exactly 32-byte long. 2. Ensure TripleO generates a key with an acceptable size. If key provided by the operator, set constraints in order to validate it at deployment time (rather than at Octavia service start up time, i.e. fail early) Either way, we might have to explicitly document what is the key size expectations and validate its content at service startup.
*** Bug 1736267 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811