Description of problem: The Load Balancer remained with ACTIVE and ONLINE state even when the listener had provisioning_status PENDING_UPDATE and the pool was with ERROR, what made the load-balancer to be considered functional by Kuryr and a load-balancer member removal was attempt without success as pool was immutable: 2021-11-03 17:04:53.893 1 ERROR kuryr_kubernetes.handlers.logging openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://10.x.x.x:13876/v2.0/lbaas/po ols/302709f7-d122-49d7-9d05-8e57ccf2ad11/members/214e892a-4e25-46ce-8bba-9ae005304931, Pool 302709f7-d122-49d7-9d05-8e57ccf2ad11 is immutable and cannot be updated. 2021-11-03 17:04:53.893 1 ERROR kuryr_kubernetes.handlers.logging (shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer show 8d03213c-ab26-4391-9d26-4878fa6b5d02 +---------------------+-------------------------------------------+ | Field | Value | +---------------------+-------------------------------------------+ | admin_state_up | True | | created_at | 2021-11-02T05:21:15 | | description | | | flavor_id | None | | id | 8d03213c-ab26-4391-9d26-4878fa6b5d02 | | listeners | 39f36082-3a07-44c0-9b00-41ba12db49fa | | name | openshift-marketplace/community-operators | | operating_status | ONLINE | | pools | 302709f7-d122-49d7-9d05-8e57ccf2ad11 | | project_id | fd0d6a21436d4ff987682c3a0419569d | | provider | ovn | | provisioning_status | ACTIVE | | updated_at | 2021-11-02T13:45:52 | | vip_address | 172.30.237.127 | | vip_network_id | f738aa91-61e1-4571-8eac-f3979164393f | | vip_port_id | 72502dfe-e6f8-4e9b-9f00-b846bcf693f8 | | vip_qos_policy_id | None | | vip_subnet_id | 14d2bae5-7de6-4f77-882b-1d5c919251fc | +---------------------+-------------------------------------------+ (shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer listener show 39f36082-3a07-44c0-9b00-41ba12db49fa +-----------------------------+-----------------------------------------------------+ | Field | Value | +-----------------------------+-----------------------------------------------------+ | admin_state_up | True | | connection_limit | -1 | | created_at | 2021-11-02T05:21:44 | | default_pool_id | 302709f7-d122-49d7-9d05-8e57ccf2ad11 | | default_tls_container_ref | None | | description | | | id | 39f36082-3a07-44c0-9b00-41ba12db49fa | | insert_headers | None | | l7policies | | | loadbalancers | 8d03213c-ab26-4391-9d26-4878fa6b5d02 | | name | openshift-marketplace/community-operators:TCP:50051 | | operating_status | ONLINE | | project_id | fd0d6a21436d4ff987682c3a0419569d | | protocol | TCP | | protocol_port | 50051 | | provisioning_status | PENDING_UPDATE | | sni_container_refs | [] | | timeout_client_data | 50000 | | timeout_member_connect | 5000 | | timeout_member_data | 50000 | | timeout_tcp_inspect | 0 | | updated_at | 2021-11-02T13:44:27 | | client_ca_tls_container_ref | None | | client_authentication | NONE | | client_crl_container_ref | None | | allowed_cidrs | None | +-----------------------------+-----------------------------------------------------+ (shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer pool show 302709f7-d122-49d7-9d05-8e57ccf2ad11 +----------------------+-----------------------------------------------------+ | Field | Value | +----------------------+-----------------------------------------------------+ | admin_state_up | True | | created_at | 2021-11-02T05:21:49 | | description | | | healthmonitor_id | | | id | 302709f7-d122-49d7-9d05-8e57ccf2ad11 | | lb_algorithm | SOURCE_IP_PORT | | listeners | 39f36082-3a07-44c0-9b00-41ba12db49fa | | loadbalancers | 8d03213c-ab26-4391-9d26-4878fa6b5d02 | | members | 214e892a-4e25-46ce-8bba-9ae005304931 | | name | openshift-marketplace/community-operators:TCP:50051 | | operating_status | OFFLINE | | project_id | fd0d6a21436d4ff987682c3a0419569d | | protocol | TCP | | provisioning_status | ERROR | | session_persistence | None | | updated_at | 2021-11-02T13:45:52 | | tls_container_ref | None | | ca_tls_container_ref | None | | crl_container_ref | None | | tls_enabled | False | +----------------------+-----------------------------------------------------+ (shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer member list 302709f7-d122-49d7-9d05-8e57ccf2ad11 +--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight | +--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ | 214e892a-4e25-46ce-8bba-9ae005304931 | openshift-marketplace/community-operators-zbmxw:50051 | fd0d6a21436d4ff987682c3a0419569d | ACTIVE | 10.128.46.128 | 50051 | NO_MONITOR | 1 | +--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+ Version-Release number of selected component (if applicable): $ cat /etc/rhosp-release Red Hat OpenStack Platform release 16.1.7 GA (Train) OCP 4.9.0-0.nightly-2021-11-01-180550 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Moving to Squad:OVN For most of the requests, the octavia API sets the provisioning status of the load balancer and its listeners to PENDING_UPDATE (this is a kind of lock on the resources) For "pool delete" for instance, it is https://opendev.org/openstack/octavia/src/branch/master/octavia/api/v2/controllers/pool.py#L512-L514 But there are some mistakes in the exception/error handling in the ovn-provider. For instance, if something occurs when deleting a pool the exception handler does: except Exception: LOG.exception(ovn_const.EXCEPTION_MSG, "deletion of pool") status = { constants.POOLS: [ {constants.ID: pool[constants.ID], constants.PROVISIONING_STATUS: constants.ERROR}], constants.LOADBALANCERS: [ {constants.ID: pool[constants.LOADBALANCER_ID], constants.PROVISIONING_STATUS: constants.ACTIVE}]} (from https://opendev.org/openstack/ovn-octavia-provider/src/branch/master/ovn_octavia_provider/helper.py#L1381-L1388) but it should also set the provisioning_status of the listener (if it exists) to ACTIVE, or the resource will be stuck with the lock. Note that the amphora driver does it at https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/v2/tasks/lifecycle_tasks.py#L193-L198
*** Bug 2044289 has been marked as a duplicate of this bug. ***
Ran 3 conformance parallel tests runs No listeners with PENDING_UPDATE listeners OSP RHOS-16.1-RHEL-8-20221122.n.1 OCP 4.10.43
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8795