Bug 2121139 - Octavia cannot configure an amphora after a FFU from 13 to 16.1
Summary: Octavia cannot configure an amphora after a FFU from 13 to 16.1
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-octavia
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z9
: 16.1 (Train on RHEL 8.2)
Assignee: Tom Weininger
QA Contact: Bruna Bonguardo
URL:
Whiteboard:
Depends On:
Blocks: 2128663
TreeView+ depends on / blocked
 
Reported: 2022-08-24 14:58 UTC by Gregory Thiemonge
Modified: 2022-12-07 20:27 UTC (History)
4 users (show)

Fixed In Version: openstack-octavia-5.0.3-1.20220927093803.8c32d2e.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2128663 (view as bug list)
Environment:
Last Closed: 2022-12-07 20:27:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack Storyboard 2010254 0 None None None 2022-08-29 12:45:48 UTC
OpenStack gerrit 854609 0 None MERGED Handle feature compatibility of HAProxy server-state-file option 2022-09-22 15:19:38 UTC
OpenStack gerrit 858506 0 None NEW Handle feature compatibility of HAProxy server-state-file option 2022-11-10 16:41:05 UTC
Red Hat Issue Tracker OSP-18364 0 None None None 2022-08-24 15:12:17 UTC
Red Hat Product Errata RHBA-2022:8795 0 None None None 2022-12-07 20:27:42 UTC

Description Gregory Thiemonge 2022-08-24 14:58:28 UTC
Description of problem:

There's a current issue with the 13->16.1 FFU, it doesn't update the octavia amphora image (BZ 2120981).
However the Octavia control plane from 16.1 should be able to configure amphora from 13.

But when upgrading to 16.1z9 requests to the amphora fail with:

2022-08-21 18:06:52.203 15 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-d8d1bbc3-b5d4-4656-8cd8-864d9009d879 - 2a9b733338d7411089a1852195a48110 - - -] request url https://172.24.0.221:9443/1.0/loadbalancer/019c24d1-d858-4272-8430-7a5773af53f1/9645c4fa-e00f-4cc3-a623-3ce9762c6be1/haproxy request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:672
2022-08-21 18:06:52.246 15 DEBUG octavia.amphorae.drivers.haproxy.rest_api_driver [req-d8d1bbc3-b5d4-4656-8cd8-864d9009d879 - 2a9b733338d7411089a1852195a48110 - - -] Connected to amphora. Response: <Response [400]> request /usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py:693
2022-08-21 18:06:52.246 15 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-d8d1bbc3-b5d4-4656-8cd8-864d9009d879 - 2a9b733338d7411089a1852195a48110 - - -] Amphora agent returned unexpected result code 400 with response {'message': 'Invalid request', 'details': "[ALERT] 232/140652 (1521) : parsing [/var/lib/octavia/9645c4fa-e00f-4cc3-a623-3ce9762c6be1/haproxy.cfg.new:8] : unknown keyword 'server-state-file' in 'global' section\n[ALERT] 232/140652 (1521) : Error(s) found in configuration file : /var/lib/octavia/9645c4fa-e00f-4cc3-a623-3ce9762c6be1/haproxy.cfg.new\n[ALERT] 232/140652 (1521) : Fatal errors found in configuration.\n"}
2022-08-21 18:06:52.249 15 WARNING octavia.controller.worker.v1.controller_worker [-] Task 'octavia.controller.worker.v1.tasks.amphora_driver_tasks.ListenersUpdate' (4f67403e-6680-421d-86df-2a02bcc53246) transitioned into state 'FAILURE' from state 'RUNNING'
5 predecessors (most recent first):
  Atom 'octavia.controller.worker.v1.tasks.model_tasks.DeleteModelObject' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'object': <octavia.common.data_models.Pool object at 0x7ff9a4549668>}, 'provides': None}
  |__Atom 'octavia.controller.worker.v1.tasks.database_tasks.CountPoolChildrenForQuota' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7ff9a4549668>}, 'provides': {'HM': 0, 'member': 0}}
     |__Atom 'octavia.controller.worker.v1.tasks.database_tasks.MarkPoolPendingDeleteInDB' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7ff9a4549668>}, 'provides': None}
        |__Atom 'octavia.controller.worker.v1.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7ff9a4549668>, 'listeners': [], 'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7ff9a4635a90>}, 'provides': None}
           |__Flow 'octavia-delete-pool-flow': octavia.amphorae.drivers.haproxy.exceptions.InvalidRequest: Invalid request
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker Traceback (most recent call last):
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker     result = task.execute(**arguments)
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 100, in execute
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker     self.amphora_driver.update(loadbalancer)
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 253, in update
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker     self.update_amphora_listeners(loadbalancer, amphora)
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 224, in update_amphora_listeners
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker     timeout_dict=timeout_dict)
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 884, in upload_config
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker     return exc.check_exception(r)
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/exceptions.py", line 44, in check_exception
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker     raise responses[status_code]()
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker octavia.amphorae.drivers.haproxy.exceptions.InvalidRequest: Invalid request
2022-08-21 18:06:52.249 15 ERROR octavia.controller.worker.v1.controller_worker 


This regression was introduced in BZ 1996756, the code added to 16.1z9 was not tested against an amphora from 13.


Version-Release number of selected component (if applicable):
16.1z9 16.2

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP13 with Octavia
2. Upgrade to OSP16.x (FFU)
3. Create a LB and configure a listener

Actual results:
It fails with the above backtrace

Comment 12 errata-xmlrpc 2022-12-07 20:27:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795


Note You need to log in before you can comment on or make changes to this bug.