Bug 1996088

Summary:	fails to delete loadbalancer with osc_lib.exceptions.CommandError: Unable to locate loadbalancers
Product:	Red Hat OpenStack	Reporter:	Gregory Thiemonge <gthiemon>
Component:	python-octaviaclient	Assignee:	Gregory Thiemonge <gthiemon>
Status:	CLOSED ERRATA	QA Contact:	Omer Schwartz <oschwart>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	asyedham, bbonguar, jelynch, konguyen, lpeer, majopela, michjohn, oschwart, scohen
Target Milestone:	z9	Keywords:	Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-octaviaclient-1.10.1-1.20220215154223.b5397ea.el8ost	Doc Type:	Bug Fix
Doc Text:	Before this update, python-octaviaclient did not display the full list of load balancers when the user had more than 1,000 load balancers. With this update, the OpenStack Load-balancing service (Octavia) displays all load balancers.	Story Points:	---
Clone Of:	1959694	Environment:
Last Closed:	2022-12-07 20:25:25 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1959694
Bug Blocks:

Description Gregory Thiemonge 2021-08-20 13:38:10 UTC

+++ This bug was initially created as a clone of Bug #1959694 +++

Description of problem:
Currently, we the Performance and Scale team are performing Octavia scale testing and we have scaled up to 2600 loadbalancers in our env using browbeat.

Scenario used: https://github.com/cloud-bulldozer/browbeat/blob/master/rally/octavia/octavia-create-show-loadbalancers.yml

MariaDB [octavia]>  select COUNT(*) from load_balancer;
+----------+
| COUNT(*) |
+----------+
|  2600    |
+----------+
1 row in set (0.002 sec)

MariaDB [octavia]>  select COUNT(provisioning_status) from load_balancer where provisioning_status = "ACTIVE";
+----------------------------+
| COUNT(provisioning_status) |
+----------------------------+
|                       2599 |
+----------------------------+
1 row in set (0.002 sec)

MariaDB [octavia]>  select COUNT(provisioning_status) from load_balancer where provisioning_status = "ERROR";
+----------------------------+
| COUNT(provisioning_status) |
+----------------------------+
|                          1 |
+----------------------------+
1 row in set (0.000 sec)

we are unable to delete the loadbalancer which was moved to the error state

(overcloud) [stack@undercloud ~]$ openstack loadbalancer list --provisioning-status ERROR
+--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+
| id                                   | name                      | project_id                       | vip_address | provisioning_status | provider |
+--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+
| c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 | s_rally_d694abc8_5UrnaqBm | 05f291f113624aefbf170f23a14f8604 | 10.2.3.184  | ERROR               | amphora  |
+--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+

(overcloud) [stack@undercloud ~]$ openstack loadbalancer delete c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 --debug
During handling of the above exception, another exception occurred:                                                                                                                                                
                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand                                                                                                                                
    result = cmd.run(parsed_args)                                                                                                                                                                                  
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/load_balancer.py", line 168, in take_action
    parsed_args)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 173, in get_loadbalancer_attrs                                                                                                      
    attrs = _map_attrs(_attrs, attr_map)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 35, in _map_attrs
    v,
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 118, in get_resource_id
    raise exceptions.CommandError(msg)
osc_lib.exceptions.CommandError: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers
clean_up DeleteLoadBalancer: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 115, in get_resource_id
    return names[0].get('id')
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/load_balancer.py", line 168, in take_action                                                                                                         
    parsed_args)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 173, in get_loadbalancer_attrs                                                                                                      
    attrs = _map_attrs(_attrs, attr_map)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 35, in _map_attrs
    v,
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 118, in get_resource_id
    raise exceptions.CommandError(msg)
osc_lib.exceptions.CommandError: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers

END return value: 1

 

Version-Release number of selected component (if applicable):
RHOS-16.1-RHEL-8-20210205.n.0

How reproducible:
100%

Steps to Reproduce:
1.Deploy OSP with Octavia
2.Scale up to 2k+ LB's


Actual results:
Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers


Expected results:
Loadbalancer should be deleted successfully

Additional info:

The ``openStack loadbalancer list`` command only fetches 1k LB's as the pagination_max_limit = 1000 is set by default, so it's not able to delete LB's whose ID is not reported by ``openstack loadbalancer list`` even though we fetched the id of the LB that was moved to error appending ``--provisioning-status ERROR`` option

Comment 17 Omer Schwartz 2022-11-15 15:43:04 UTC

I was able to create 2600 LB, and all of those were deployed with ACTIVE provisioning-status:

(overcloud) [stack@undercloud-0 ~]$ cat core_puddle_version 
RHOS-16.1-RHEL-8-20221108.n.1

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list --provisioning-status ACTIVE -f value | wc -l
2600
^
I could list all of the 2600

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list --provisioning-status ERROR -f value | wc -l
0



All LBs were deleted without any error:
(overcloud) [stack@undercloud-0 ~]$ for lb in $(openstack loadbalancer list --provisioning-status ACTIVE -f value | awk '{print $1}'); do openstack loadbalancer delete $lb; done

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list --provisioning-status ACTIVE -f value | wc -l
0

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list --provisioning-status ERROR -f value | wc -l
0

Looks good to me. I am moving the BZ status to VERIFIED.

Comment 24 errata-xmlrpc 2022-12-07 20:25:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8795