Bug 2019980

Summary: Load Balancer remained with ACTIVE state even with PENDING_UPDATE listener
Product: Red Hat OpenStack Reporter: Maysa Macedo <mdemaced>
Component: python-networking-ovnAssignee: Luis Tomas Bolivar <ltomasbo>
Status: CLOSED ERRATA QA Contact: Omer Schwartz <oschwart>
Severity: high Docs Contact:
Priority: high    
Version: 16.1 (Train)CC: apevec, egarciar, ekuris, froyo, gthiemon, itbrown, jlibosva, juzhao, lhh, ltomasbo, majopela, mdulko, scohen, ykarel
Target Milestone: z9Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-networking-ovn-7.3.1-1.20220125210409.4e24f4c.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2045088 (view as bug list) Environment:
Last Closed: 2022-12-07 20:25:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2045088    
Bug Blocks:    

Description Maysa Macedo 2021-11-03 18:40:23 UTC 
Description of problem:

The Load Balancer remained with ACTIVE and ONLINE state even when the listener had provisioning_status PENDING_UPDATE and the pool was with ERROR, what made the load-balancer to be considered functional by Kuryr and a load-balancer member removal was attempt without success as pool was immutable:

2021-11-03 17:04:53.893 1 ERROR kuryr_kubernetes.handlers.logging openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://10.x.x.x:13876/v2.0/lbaas/po
ols/302709f7-d122-49d7-9d05-8e57ccf2ad11/members/214e892a-4e25-46ce-8bba-9ae005304931, Pool 302709f7-d122-49d7-9d05-8e57ccf2ad11 is immutable and cannot be updated.                          
2021-11-03 17:04:53.893 1 ERROR kuryr_kubernetes.handlers.logging 

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer show 8d03213c-ab26-4391-9d26-4878fa6b5d02     
+---------------------+-------------------------------------------+
| Field               | Value                                     |
+---------------------+-------------------------------------------+
| admin_state_up      | True                                      |
| created_at          | 2021-11-02T05:21:15                       |
| description         |                                           |
| flavor_id           | None                                      |
| id                  | 8d03213c-ab26-4391-9d26-4878fa6b5d02      |
| listeners           | 39f36082-3a07-44c0-9b00-41ba12db49fa      |
| name                | openshift-marketplace/community-operators |
| operating_status    | ONLINE                                    |
| pools               | 302709f7-d122-49d7-9d05-8e57ccf2ad11      |
| project_id          | fd0d6a21436d4ff987682c3a0419569d          |
| provider            | ovn                                       |
| provisioning_status | ACTIVE                                    |
| updated_at          | 2021-11-02T13:45:52                       |
| vip_address         | 172.30.237.127                            |
| vip_network_id      | f738aa91-61e1-4571-8eac-f3979164393f      |
| vip_port_id         | 72502dfe-e6f8-4e9b-9f00-b846bcf693f8      |
| vip_qos_policy_id   | None                                      |
| vip_subnet_id       | 14d2bae5-7de6-4f77-882b-1d5c919251fc      |
+---------------------+-------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer listener show 39f36082-3a07-44c0-9b00-41ba12db49fa
+-----------------------------+-----------------------------------------------------+
| Field                       | Value                                               |
+-----------------------------+-----------------------------------------------------+
| admin_state_up              | True                                                |
| connection_limit            | -1                                                  |
| created_at                  | 2021-11-02T05:21:44                                 |
| default_pool_id             | 302709f7-d122-49d7-9d05-8e57ccf2ad11                |
| default_tls_container_ref   | None                                                |
| description                 |                                                     |
| id                          | 39f36082-3a07-44c0-9b00-41ba12db49fa                |
| insert_headers              | None                                                |
| l7policies                  |                                                     |
| loadbalancers               | 8d03213c-ab26-4391-9d26-4878fa6b5d02                |
| name                        | openshift-marketplace/community-operators:TCP:50051 |
| operating_status            | ONLINE                                              |
| project_id                  | fd0d6a21436d4ff987682c3a0419569d                    |
| protocol                    | TCP                                                 |
| protocol_port               | 50051                                               |
| provisioning_status         | PENDING_UPDATE                                      |
| sni_container_refs          | []                                                  |
| timeout_client_data         | 50000                                               |
| timeout_member_connect      | 5000                                                |
| timeout_member_data         | 50000                                               |
| timeout_tcp_inspect         | 0                                                   |
| updated_at                  | 2021-11-02T13:44:27                                 |
| client_ca_tls_container_ref | None                                                |
| client_authentication       | NONE                                                |
| client_crl_container_ref    | None                                                |
| allowed_cidrs               | None                                                |
+-----------------------------+-----------------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer pool show 302709f7-d122-49d7-9d05-8e57ccf2ad11
+----------------------+-----------------------------------------------------+
| Field                | Value                                               |
+----------------------+-----------------------------------------------------+
| admin_state_up       | True                                                |
| created_at           | 2021-11-02T05:21:49                                 |
| description          |                                                     |
| healthmonitor_id     |                                                     |
| id                   | 302709f7-d122-49d7-9d05-8e57ccf2ad11                |
| lb_algorithm         | SOURCE_IP_PORT                                      |
| listeners            | 39f36082-3a07-44c0-9b00-41ba12db49fa                |
| loadbalancers        | 8d03213c-ab26-4391-9d26-4878fa6b5d02                |
| members              | 214e892a-4e25-46ce-8bba-9ae005304931                |
| name                 | openshift-marketplace/community-operators:TCP:50051 |
| operating_status     | OFFLINE                                             |
| project_id           | fd0d6a21436d4ff987682c3a0419569d                    |
| protocol             | TCP                                                 |
| provisioning_status  | ERROR                                               |
| session_persistence  | None                                                |
| updated_at           | 2021-11-02T13:45:52                                 |
| tls_container_ref    | None                                                |
| ca_tls_container_ref | None                                                |
| crl_container_ref    | None                                                |
| tls_enabled          | False                                               |
+----------------------+-----------------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer member list 302709f7-d122-49d7-9d05-8e57ccf2ad11
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+
| id                                   | name                                                  | project_id                       | provisioning_status | address       | protocol_port | operating_status | weight |
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+
| 214e892a-4e25-46ce-8bba-9ae005304931 | openshift-marketplace/community-operators-zbmxw:50051 | fd0d6a21436d4ff987682c3a0419569d | ACTIVE              | 10.128.46.128 |         50051 | NO_MONITOR       |      1 |
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+

Version-Release number of selected component (if applicable):
$ cat /etc/rhosp-release 
Red Hat OpenStack Platform release 16.1.7 GA (Train)
OCP 4.9.0-0.nightly-2021-11-01-180550

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Gregory Thiemonge 2021-11-08 10:15:29 UTC 
Moving to Squad:OVN


For most of the requests, the octavia API sets the provisioning status of the load balancer and its listeners to PENDING_UPDATE (this is a kind of lock on the resources)

For "pool delete" for instance, it is https://opendev.org/openstack/octavia/src/branch/master/octavia/api/v2/controllers/pool.py#L512-L514


But there are some mistakes in the exception/error handling in the ovn-provider.

For instance, if something occurs when deleting a pool the exception handler does:

        except Exception:
            LOG.exception(ovn_const.EXCEPTION_MSG, "deletion of pool")
            status = {
                constants.POOLS: [
                    {constants.ID: pool[constants.ID],
                     constants.PROVISIONING_STATUS: constants.ERROR}],
                constants.LOADBALANCERS: [
                    {constants.ID: pool[constants.LOADBALANCER_ID],
                     constants.PROVISIONING_STATUS: constants.ACTIVE}]}

(from https://opendev.org/openstack/ovn-octavia-provider/src/branch/master/ovn_octavia_provider/helper.py#L1381-L1388)

but it should also set the provisioning_status of the listener (if it exists) to ACTIVE, or the resource will be stuck with the lock.

Note that the amphora driver does it at https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/v2/tasks/lifecycle_tasks.py#L193-L198
Comment 6 Michał Dulko 2022-02-08 15:50:56 UTC 
*** Bug 2044289 has been marked as a duplicate of this bug. ***
Comment 18 Itzik Brown 2022-11-24 20:40:04 UTC 
Ran 3 conformance parallel tests runs
No listeners with PENDING_UPDATE listeners

OSP RHOS-16.1-RHEL-8-20221122.n.1
OCP 4.10.43
Comment 24 errata-xmlrpc 2022-12-07 20:25:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795