Bug 1668369
| Summary: | Creating listener with HTTP protocol and pool with TCP logs with errors | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Stafeyev <astafeye> |
| Component: | openstack-octavia | Assignee: | Carlos Goncalves <cgoncalves> |
| Status: | CLOSED ERRATA | QA Contact: | Bruna Bonguardo <bbonguar> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 14.0 (Rocky) | CC: | amuller, astafeye, cgoncalves, ihrachys, lpeer, majopela, mgarciac, nmanos |
| Target Milestone: | z11 | Keywords: | Triaged, ZStream |
| Target Release: | 13.0 (Queens) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-octavia-2.1.2-4.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-06-24 11:53:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** Bug 1573497 has been marked as a duplicate of this bug. *** *** Bug 1749277 has been marked as a duplicate of this bug. *** OSP 14 ELS is in ~1 month. Instead of closing, I'm changing the target release since this is also relevant for OSP 13. Patch in upstream master (Ussuri development cycle) has merged. Pool create with invalid protocol combinations is now errored out at API level. Feasibility of backporting to stable branches is being discussed as it may violate the stable branch support policy. $ openstack loadbalancer create --vip-subnet-id private-subnet --name lb1 +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | availability_zone | None | | created_at | 2020-01-09T09:04:26 | | description | | | flavor_id | None | | id | 42f7ce58-308b-44c9-acd7-3e33e9c83b2f | | listeners | | | name | lb1 | | operating_status | OFFLINE | | pools | | | project_id | 6035c5306a37489399d5dd9c6d7ba033 | | provider | amphora | | provisioning_status | PENDING_CREATE | | updated_at | None | | vip_address | 10.0.0.38 | | vip_network_id | 7735cc5a-2695-4b0b-8343-ae1a77392412 | | vip_port_id | 189e54be-0e85-486a-9fa3-68ec714e385a | | vip_qos_policy_id | None | | vip_subnet_id | b4b209ac-6681-4046-8f94-27a8a6d2d560 | +---------------------+--------------------------------------+ $ openstack loadbalancer listener create --name listener1 --protocol HTTP --protocol-port 80 lb1 +-----------------------------+--------------------------------------+ | Field | Value | +-----------------------------+--------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2020-01-09T10:01:25 | | default_pool_id | None | | default_tls_container_ref | None | | description | | | id | a167b3e5-6fa1-44ed-a509-e9c927b2c290 | | insert_headers | None | | l7policies | | | loadbalancers | 42f7ce58-308b-44c9-acd7-3e33e9c83b2f | | name | listener1 | | operating_status | OFFLINE | | project_id | 6035c5306a37489399d5dd9c6d7ba033 | | protocol | HTTP | | protocol_port | 80 | | provisioning_status | PENDING_CREATE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | None | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+--------------------------------------+ $ openstack loadbalancer pool create --protocol TCP --lb-algorithm ROUND_ROBIN --listener listener1 --name pool1 Validation failure: The pool protocol 'TCP' is invalid while the listener protocol is 'HTTP'. (HTTP 400) (Request-ID: req-fdb470b6-cb6f-4eab-b07d-f9211ccb2e39) Protocol combination matrix: https://docs.openstack.org/api-ref/load-balancer/v2/index.html#protocol-combinations Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2724 |
Description of problem: Create listener http port 80 and pool TCP. Saw errors in log and pool in error state: (test_cloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener show 7732f336-4040-4870-975f-22663e6a3f19 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2019-01-22T14:53:25 | | default_pool_id | None | | default_tls_container_ref | None | | description | | | id | 7732f336-4040-4870-975f-22663e6a3f19 | | insert_headers | None | | l7policies | | | loadbalancers | 04fbea00-4ea1-40bd-a56d-2a519986a91f | | name | | | operating_status | ONLINE | | project_id | 20702966a73243c788964cc3c7164282 | | protocol | HTTP | | protocol_port | 80 | | provisioning_status | ACTIVE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2019-01-22T14:53:32 | +---------------------------+--------------------------------------+ (test_cloud) [stack@undercloud-0 ~]$ openstack loadbalancer pool show 4eb1c201-9856-4c74-a75b-77e29f9c442c +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2019-01-22T14:55:54 | | description | | | healthmonitor_id | | | id | 4eb1c201-9856-4c74-a75b-77e29f9c442c | | lb_algorithm | ROUND_ROBIN | | listeners | 7732f336-4040-4870-975f-22663e6a3f19 | | loadbalancers | 04fbea00-4ea1-40bd-a56d-2a519986a91f | | members | | | name | | | operating_status | OFFLINE | | project_id | 20702966a73243c788964cc3c7164282 | | protocol | TCP | | provisioning_status | ERROR | | session_persistence | None | | updated_at | 2019-01-22T14:55:55 | +---------------------+--------------------------------------+ (test_cloud) [stack@undercloud-0 ~]$ Version-Release number of selected component (if applicable): 14 (test_cloud) [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 14 -p 2019-01-17.2 (test_cloud) [stack@undercloud-0 ~]$ Steps to Reproduce: 1. create LB in internal network 2. Create listener as mentioned 3. Create Pool as mentioned Actual results: Poll created in error state Expected results: pool created properly OR prevention of pool creation with provided information to the user. Additional info: [root@controller-2 ~]# tailf /var/log/containers/octavia/worker.log 2019-01-22 14:55:54.622 22 INFO octavia.controller.queue.endpoin t [-] Creating pool '4eb1c201-9856-4c74-a75b-77e29f9c442c'... 2019-01-22 14:55:55.309 22 ERROR octavia.amphorae.drivers.haproxy.exceptions [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Amphora agent returned unexpected result code 400 with response {u'message': u'Invalid request', u'details': u"[ALERT] 021/095554 (3795) : Unable to use proxy '4eb1c201-9856-4c74-a75b-77e29f9c442c' with wrong mode, required: http, has: tcp.\n[ALERT] 021/095554 (3795) : You may want to use 'mode http'.\n[ALERT] 021/095554 (3795) : Proxy '7732f336-4040-4870-975f-22663e6a3f19': unable to find required default_backend: '4eb1c201-9856-4c74-a75b-77e29f9c442c'.\n[ALERT] 021/095554 (3795) : Fatal errors found in configuration.\n"} 2019-01-22 14:55:55.314 22 WARNING octavia.controller.worker.controller_worker [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenersUpdate' (20f8cf6b-8cf3-4f6c-8776-a510bfc00ebe) transitioned into state 'FAILURE' from state 'RUNNING' 3 predecessors (most recent first): Atom 'octavia.controller.worker.tasks.database_tasks.MarkPoolPendingCreateInDB' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7f36fe54d450>}, 'provides': None} |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'listeners': [<octavia.common.data_models.Listener object at 0x7f36f2555f50>], 'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7f36f2566d50>, 'pool': <octavia.common.data_models.Pool object at 0x7f36fe54d450>}, 'provides': None} |__Flow 'octavia-create-pool-flow': InvalidRequest: Invalid request 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last): 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker result = task.execute(**arguments) 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 75, in execute 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker self.amphora_driver.update(listener, loadbalancer.vip) 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 143, in update 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker self.client.upload_config(amp, listener.id, config) 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 400, in upload_config 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker return exc.check_exception(r) 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/exceptions.py", line 43, in check_exception 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker raise responses[status_code]() 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker InvalidRequest: Invalid request 2019-01-22 14:55:55.314 22 ERROR octavia.controller.worker.controller_worker 2019-01-22 14:55:55.319 22 WARNING octavia.controller.worker.tasks.amphora_driver_tasks [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Reverting listeners updates. 2019-01-22 14:55:55.335 22 WARNING octavia.controller.worker.controller_worker [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenersUpdate' (20f8cf6b-8cf3-4f6c-8776-a510bfc00ebe) transitioned into state 'REVERTED' from state 'REVERTING' 2019-01-22 14:55:55.337 22 WARNING octavia.controller.worker.tasks.database_tasks [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Reverting mark pool pending create in DB for pool id 4eb1c201-9856-4c74-a75b-77e29f9c442c 2019-01-22 14:55:55.344 22 WARNING octavia.controller.worker.controller_worker [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Task 'octavia.controller.worker.tasks.database_tasks.MarkPoolPendingCreateInDB' (4628f036-a996-47ef-8719-efe8a439ec59) transitioned into state 'REVERTED' from state 'REVERTING' 2019-01-22 14:55:55.373 22 WARNING octavia.controller.worker.controller_worker [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Task 'octavia.controller.worker.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' (66ac99fb-fb34-4f2f-87e6-ba2a09f5fe87) transitioned into state 'REVERTED' from state 'REVERTING' 2019-01-22 14:55:55.376 22 WARNING octavia.controller.worker.controller_worker [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Flow 'octavia-create-pool-flow' (20111123-ae37-428e-90a0-d91c31c6d91e) transitioned into state 'REVERTED' from state 'RUNNING' 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server [req-71d021e0-0a46-4d5a-83e5-4115ab0cc469 - 20702966a73243c788964cc3c7164282 - - -] Exception during message handling: InvalidRequest: Invalid request 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/octavia/controller/queue/endpoint.py", line 80, in create_pool 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server self.worker.create_pool(pool_id) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 241, in wrapped_f 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server return self.call(f, *args, **kw) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 330, in call 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server start_time=start_time) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 279, in iter 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server return fut.result() 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server return self.__get_result() 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 333, in call 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server result = fn(*args, **kwargs) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/octavia/controller/worker/controller_worker.py", line 571, in create_pool 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server create_pool_tf.run() 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server failures[0].reraise() 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 346, in reraise 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server result = task.execute(**arguments) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 75, in execute 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server self.amphora_driver.update(listener, loadbalancer.vip) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 143, in update 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server self.client.upload_config(amp, listener.id, config) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 400, in upload_config 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server return exc.check_exception(r) 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/exceptions.py", line 43, in check_exception 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server raise responses[status_code]() 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server InvalidRequest: Invalid request 2019-01-22 14:55:55.376 22 ERROR oslo_messaging.rpc.server