1800847 – Octavia controller services fail to load CA certificate

Bug 1800847 - Octavia controller services fail to load CA certificate

Summary: Octavia controller services fail to load CA certificate

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-common
Sub Component:
Version:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	z11
Target Release:	13.0 (Queens)
Assignee:	Carlos Goncalves
QA Contact:	Gregory Thiemonge
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1688323 1755683 1756474 1759254
TreeView+	depends on / blocked

Reported:	2020-02-08 10:51 UTC by Carlos Goncalves
Modified:	2020-03-10 11:24 UTC (History)
CC List:	4 users (show)
Fixed In Version:	openstack-tripleo-common-8.7.1-11.el7ost
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-03-10 11:24:20 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1830190	None	None	None	2020-02-08 10:55:51 UTC
OpenStack gerrit	706634	None	MERGED	Fix Octavia certificate file path and content	2020-10-25 20:00:34 UTC
Red Hat Product Errata	RHBA-2020:0760	None	None	None	2020-03-10 11:24:51 UTC

Description Carlos Goncalves 2020-02-08 10:51:45 UTC

An error occurs when creating a load balancer. The Octavia Worker cannot load the CA certificate. This is a regression introduced in preparation for z11.

Environment: OSP 13 2020-02-06.2


2020-02-07 21:01:03.426 23 INFO octavia.controller.queue.endpoint [-] Creating load balancer '1cbde451-b9fa-4946-82da-acf2ef3ad695'...
2020-02-07 21:01:04.568 23 INFO octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB with id 5483a1eb-9c09-46e7-b1b3-be8e2233867b
2020-02-07 21:01:05.099 23 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
2020-02-07 21:01:05.100 23 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
2020-02-07 21:01:05.105 23 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' (e3e5277e-8beb-4f18-a333-8299747ffa10) transitioned into state 'FAILURE' from state 'RUNNING'
6 predecessors (most recent first):
  Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': u'5483a1eb-9c09-46e7-b1b3-be8e2233867b'}
  |__Flow 'STANDALONE-octavia-create-amp-for-lb-subflow'
     |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'1cbde451-b9fa-4946-82da-acf2ef3ad695'}, 'provides': None}
        |__Flow 'STANDALONE-octavia-get-amphora-for-lb-subflow'
           |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'1cbde451-b9fa-4946-82da-acf2ef3ad695'}, 'provides': None}
              |__Flow 'octavia-create-loadbalancer-flow': CertificateGenerationException: Could not sign the certificate request: Failed to load CA Certificate /etc/octavia/certs/ca_01.pem.
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker     result = task.execute(**arguments)
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/cert_task.py", line 48, in execute
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker     validity=CERT_VALIDITY)
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/site-packages/octavia/certificates/generator/local.py", line 234, in generate_cert_key_pair
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker     cert = cls.sign_cert(csr, validity, **kwargs)
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/site-packages/octavia/certificates/generator/local.py", line 91, in sign_cert
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker     cls._validate_cert(ca_cert, ca_key, ca_key_pass)
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/site-packages/octavia/certificates/generator/local.py", line 53, in _validate_cert
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker     .format(CONF.certificates.ca_certificate)
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker CertificateGenerationException: Could not sign the certificate request: Failed to load CA Certificate /etc/octavia/certs/ca_01.pem.
2020-02-07 21:01:05.105 23 ERROR octavia.controller.worker.controller_worker
2020-02-07 21:01:05.121 23 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-generate-serverpem' (e3e5277e-8beb-4f18-a333-8299747ffa10) transitioned into state 'REVERTED' from state 'REVERTING'
2020-02-07 21:01:05.123 23 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting create amphora in DB for amp id 5483a1eb-9c09-46e7-b1b3-be8e2233867b
2020-02-07 21:01:05.186 23 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-create-amphora-indb' (d2290d52-a130-45b4-8b88-bc0093c1a50a) transitioned into state 'REVERTED' from state 'REVERTING'
2020-02-07 21:01:05.190 23 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting Amphora allocation for the load balancer 1cbde451-b9fa-4946-82da-acf2ef3ad695 in the database.
2020-02-07 21:01:05.213 23 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' (809f3edb-d8f4-4c58-94c6-472134a99324) transitioned into state 'REVERTED' from state 'REVERTING'
2020-02-07 21:01:05.253 23 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' (5ee0b29b-d5a7-44d4-a6ff-6bdee2f32771) transitioned into state 'REVERTED' from state 'REVERTING'
2020-02-07 21:01:05.271 23 WARNING octavia.controller.worker.controller_worker [-] Flow 'octavia-create-loadbalancer-flow' (f992c185-f952-49f5-8ebd-936b7ca5d0fc) transitioned into state 'REVERTED' from state 'RUNNING'
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server [-] Exception during message handling: CertificateGenerationException: Could not sign the certificate request: Failed to load CA Certificate /etc/octavia/certs/ca_01.pem.
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/queue/endpoint.py", line 44, in create_load_balancer
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     self.worker.create_load_balancer(load_balancer_id)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/controller_worker.py", line 268, in create_load_balancer
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     create_lb_tf.run()
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     failures[0].reraise()
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     six.reraise(*self._exc_info)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/cert_task.py", line 48, in execute
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     validity=CERT_VALIDITY)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/certificates/generator/local.py", line 234, in generate_cert_key_pair
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     cert = cls.sign_cert(csr, validity, **kwargs)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/certificates/generator/local.py", line 91, in sign_cert
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     cls._validate_cert(ca_cert, ca_key, ca_key_pass)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/certificates/generator/local.py", line 53, in _validate_cert
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server     .format(CONF.certificates.ca_certificate)
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server CertificateGenerationException: Could not sign the certificate request: Failed to load CA Certificate /etc/octavia/certs/ca_01.pem.
2020-02-07 21:01:05.272 23 ERROR oslo_messaging.rpc.server

Comment 6 Bruna Bonguardo 2020-02-12 12:23:50 UTC

Waiting for openstack-tripleo-common-8.7.1-11.el7ost to be part of the osp13's passed_phase1 puddle.

For now the package in the passed_phase1 puddle is openstack-tripleo-common-8.7.1-8.el7ost.noarch.rpm, as seen in:
http://download.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/13.0-RHEL-7/2020-02-10.8/RH7-RHOS-13.0/source/

Comment 15 Gregory Thiemonge 2020-02-14 17:17:28 UTC

Using OSP 13 2020-02-14.1

Verified in controller logs that certificates can be read, then the load balancer is marked as "ACTIVE".

octavia worker logs:

2020-02-14 16:19:19.268 25 INFO octavia.controller.queue.endpoint [-] Creating load balancer 'f96147ed-eaa1-4168-9b16-fe250544d455'...
2020-02-14 16:19:19.874 25 INFO octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB with id e4495060-d330-486d-ad5d-237f9318d015
2020-02-14 16:19:19.950 25 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
2020-02-14 16:19:19.950 25 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
2020-02-14 16:19:19.951 25 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
2020-02-14 16:19:19.951 25 INFO octavia.certificates.generator.local [-] Using CA Private Key Passphrase from config.
2020-02-14 16:20:05.335 25 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectionError: HTTPSConnectionPool(host='172.24.0.7', port=
9443): Max retries exceeded with url: /0.5/info (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9723232750>: Failed to establish a n
ew connection: [Errno 113] No route to host',))
[...]
2020-02-14 16:20:44.554 25 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for amphora: e4495060-d330-486d-ad5d-237f9318d015 with compute id 8e5d278d-f449-4cde-b4
20-5f9a9d774727 for load balancer: f96147ed-eaa1-4168-9b16-fe250544d455
2020-02-14 16:20:46.156 25 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Port e77a0e2b-e84f-4aa3-b75d-095fbe9e6f0a already exists. Nothing to be done.
2020-02-14 16:20:58.914 25 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ACTIVE in DB for load balancer id: f96147ed-eaa1-4168-9b16-fe250544d455

Comment 18 errata-xmlrpc 2020-03-10 11:24:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0760

Note You need to log in before you can comment on or make changes to this bug.