Bug 2019980 - Load Balancer remained with ACTIVE state even with PENDING_UPDATE listener
Summary: Load Balancer remained with ACTIVE state even with PENDING_UPDATE listener
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z9
: 16.1 (Train on RHEL 8.2)
Assignee: Luis Tomas Bolivar
QA Contact: Omer Schwartz
URL:
Whiteboard:
: 2044289 (view as bug list)
Depends On: 2045088
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-03 18:40 UTC by Maysa Macedo
Modified: 2022-12-07 20:25 UTC (History)
14 users (show)

Fixed In Version: python-networking-ovn-7.3.1-1.20220125210409.4e24f4c.el8ost
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 2045088 (view as bug list)
Environment:
Last Closed: 2022-12-07 20:25:25 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 823544 0 None MERGED Set listeners back to ACTIVE upon pool/member action failures 2022-01-31 16:59:35 UTC
OpenStack gerrit 825215 0 None MERGED Set listeners back to ACTIVE upon pool/member action failures 2022-01-31 16:59:38 UTC
OpenStack gerrit 825303 0 None MERGED Set listeners back to ACTIVE upon pool/member action failures 2022-01-31 16:59:39 UTC
Red Hat Issue Tracker OSP-10643 0 None None None 2021-11-15 12:29:57 UTC
Red Hat Product Errata RHBA-2022:8795 0 None None None 2022-12-07 20:25:41 UTC

Description Maysa Macedo 2021-11-03 18:40:23 UTC
Description of problem:

The Load Balancer remained with ACTIVE and ONLINE state even when the listener had provisioning_status PENDING_UPDATE and the pool was with ERROR, what made the load-balancer to be considered functional by Kuryr and a load-balancer member removal was attempt without success as pool was immutable:

2021-11-03 17:04:53.893 1 ERROR kuryr_kubernetes.handlers.logging openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://10.x.x.x:13876/v2.0/lbaas/po
ols/302709f7-d122-49d7-9d05-8e57ccf2ad11/members/214e892a-4e25-46ce-8bba-9ae005304931, Pool 302709f7-d122-49d7-9d05-8e57ccf2ad11 is immutable and cannot be updated.                          
2021-11-03 17:04:53.893 1 ERROR kuryr_kubernetes.handlers.logging 

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer show 8d03213c-ab26-4391-9d26-4878fa6b5d02     
+---------------------+-------------------------------------------+
| Field               | Value                                     |
+---------------------+-------------------------------------------+
| admin_state_up      | True                                      |
| created_at          | 2021-11-02T05:21:15                       |
| description         |                                           |
| flavor_id           | None                                      |
| id                  | 8d03213c-ab26-4391-9d26-4878fa6b5d02      |
| listeners           | 39f36082-3a07-44c0-9b00-41ba12db49fa      |
| name                | openshift-marketplace/community-operators |
| operating_status    | ONLINE                                    |
| pools               | 302709f7-d122-49d7-9d05-8e57ccf2ad11      |
| project_id          | fd0d6a21436d4ff987682c3a0419569d          |
| provider            | ovn                                       |
| provisioning_status | ACTIVE                                    |
| updated_at          | 2021-11-02T13:45:52                       |
| vip_address         | 172.30.237.127                            |
| vip_network_id      | f738aa91-61e1-4571-8eac-f3979164393f      |
| vip_port_id         | 72502dfe-e6f8-4e9b-9f00-b846bcf693f8      |
| vip_qos_policy_id   | None                                      |
| vip_subnet_id       | 14d2bae5-7de6-4f77-882b-1d5c919251fc      |
+---------------------+-------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer listener show 39f36082-3a07-44c0-9b00-41ba12db49fa
+-----------------------------+-----------------------------------------------------+
| Field                       | Value                                               |
+-----------------------------+-----------------------------------------------------+
| admin_state_up              | True                                                |
| connection_limit            | -1                                                  |
| created_at                  | 2021-11-02T05:21:44                                 |
| default_pool_id             | 302709f7-d122-49d7-9d05-8e57ccf2ad11                |
| default_tls_container_ref   | None                                                |
| description                 |                                                     |
| id                          | 39f36082-3a07-44c0-9b00-41ba12db49fa                |
| insert_headers              | None                                                |
| l7policies                  |                                                     |
| loadbalancers               | 8d03213c-ab26-4391-9d26-4878fa6b5d02                |
| name                        | openshift-marketplace/community-operators:TCP:50051 |
| operating_status            | ONLINE                                              |
| project_id                  | fd0d6a21436d4ff987682c3a0419569d                    |
| protocol                    | TCP                                                 |
| protocol_port               | 50051                                               |
| provisioning_status         | PENDING_UPDATE                                      |
| sni_container_refs          | []                                                  |
| timeout_client_data         | 50000                                               |
| timeout_member_connect      | 5000                                                |
| timeout_member_data         | 50000                                               |
| timeout_tcp_inspect         | 0                                                   |
| updated_at                  | 2021-11-02T13:44:27                                 |
| client_ca_tls_container_ref | None                                                |
| client_authentication       | NONE                                                |
| client_crl_container_ref    | None                                                |
| allowed_cidrs               | None                                                |
+-----------------------------+-----------------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer pool show 302709f7-d122-49d7-9d05-8e57ccf2ad11
+----------------------+-----------------------------------------------------+
| Field                | Value                                               |
+----------------------+-----------------------------------------------------+
| admin_state_up       | True                                                |
| created_at           | 2021-11-02T05:21:49                                 |
| description          |                                                     |
| healthmonitor_id     |                                                     |
| id                   | 302709f7-d122-49d7-9d05-8e57ccf2ad11                |
| lb_algorithm         | SOURCE_IP_PORT                                      |
| listeners            | 39f36082-3a07-44c0-9b00-41ba12db49fa                |
| loadbalancers        | 8d03213c-ab26-4391-9d26-4878fa6b5d02                |
| members              | 214e892a-4e25-46ce-8bba-9ae005304931                |
| name                 | openshift-marketplace/community-operators:TCP:50051 |
| operating_status     | OFFLINE                                             |
| project_id           | fd0d6a21436d4ff987682c3a0419569d                    |
| protocol             | TCP                                                 |
| provisioning_status  | ERROR                                               |
| session_persistence  | None                                                |
| updated_at           | 2021-11-02T13:45:52                                 |
| tls_container_ref    | None                                                |
| ca_tls_container_ref | None                                                |
| crl_container_ref    | None                                                |
| tls_enabled          | False                                               |
+----------------------+-----------------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer member list 302709f7-d122-49d7-9d05-8e57ccf2ad11
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+
| id                                   | name                                                  | project_id                       | provisioning_status | address       | protocol_port | operating_status | weight |
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+
| 214e892a-4e25-46ce-8bba-9ae005304931 | openshift-marketplace/community-operators-zbmxw:50051 | fd0d6a21436d4ff987682c3a0419569d | ACTIVE              | 10.128.46.128 |         50051 | NO_MONITOR       |      1 |
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+

Version-Release number of selected component (if applicable):
$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 16.1.7 GA (Train)
OCP 4.9.0-0.nightly-2021-11-01-180550

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Gregory Thiemonge 2021-11-08 10:15:29 UTC
Moving to Squad:OVN


For most of the requests, the octavia API sets the provisioning status of the load balancer and its listeners to PENDING_UPDATE (this is a kind of lock on the resources)

For "pool delete" for instance, it is https://opendev.org/openstack/octavia/src/branch/master/octavia/api/v2/controllers/pool.py#L512-L514


But there are some mistakes in the exception/error handling in the ovn-provider.

For instance, if something occurs when deleting a pool the exception handler does:

        except Exception:
            LOG.exception(ovn_const.EXCEPTION_MSG, "deletion of pool")
            status = {
                constants.POOLS: [
                    {constants.ID: pool[constants.ID],
                     constants.PROVISIONING_STATUS: constants.ERROR}],
                constants.LOADBALANCERS: [
                    {constants.ID: pool[constants.LOADBALANCER_ID],
                     constants.PROVISIONING_STATUS: constants.ACTIVE}]}

(from https://opendev.org/openstack/ovn-octavia-provider/src/branch/master/ovn_octavia_provider/helper.py#L1381-L1388)

but it should also set the provisioning_status of the listener (if it exists) to ACTIVE, or the resource will be stuck with the lock.

Note that the amphora driver does it at https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/v2/tasks/lifecycle_tasks.py#L193-L198

Comment 6 Michał Dulko 2022-02-08 15:50:56 UTC
*** Bug 2044289 has been marked as a duplicate of this bug. ***

Comment 18 Itzik Brown 2022-11-24 20:40:04 UTC
Ran 3 conformance parallel tests runs
No listeners with PENDING_UPDATE listeners

OSP RHOS-16.1-RHEL-8-20221122.n.1
OCP 4.10.43

Comment 24 errata-xmlrpc 2022-12-07 20:25:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795


Note You need to log in before you can comment on or make changes to this bug.