Bug 1959694

Summary:	fails to delete loadbalancer with osc_lib.exceptions.CommandError: Unable to locate loadbalancers
Product:	Red Hat OpenStack	Reporter:	Asma Syed Hameed <asyedham>
Component:	python-octaviaclient	Assignee:	Gregory Thiemonge <gthiemon>
Status:	CLOSED ERRATA	QA Contact:	Bruna Bonguardo <bbonguar>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	ihrachys, konguyen, lpeer, majopela, michjohn, oschwart, scohen
Target Milestone:	z3	Keywords:	Triaged
Target Release:	16.2 (Train on RHEL 8.4)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-octaviaclient-1.10.1-2.20220215145235.b5397ea.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1996088 (view as bug list)		Environment:
Last Closed:	2022-03-23 22:10:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1996088

Description Asma Syed Hameed 2021-05-12 06:37:34 UTC

Description of problem:
Currently, we the Performance and Scale team are performing Octavia scale testing and we have scaled up to 2600 loadbalancers in our env using browbeat.

Scenario used: https://github.com/cloud-bulldozer/browbeat/blob/master/rally/octavia/octavia-create-show-loadbalancers.yml

MariaDB [octavia]>  select COUNT(*) from load_balancer;
+----------+
| COUNT(*) |
+----------+
|  2600    |
+----------+
1 row in set (0.002 sec)

MariaDB [octavia]>  select COUNT(provisioning_status) from load_balancer where provisioning_status = "ACTIVE";
+----------------------------+
| COUNT(provisioning_status) |
+----------------------------+
|                       2599 |
+----------------------------+
1 row in set (0.002 sec)

MariaDB [octavia]>  select COUNT(provisioning_status) from load_balancer where provisioning_status = "ERROR";
+----------------------------+
| COUNT(provisioning_status) |
+----------------------------+
|                          1 |
+----------------------------+
1 row in set (0.000 sec)

we are unable to delete the loadbalancer which was moved to the error state

(overcloud) [stack@undercloud ~]$ openstack loadbalancer list --provisioning-status ERROR
+--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+
| id                                   | name                      | project_id                       | vip_address | provisioning_status | provider |
+--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+
| c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 | s_rally_d694abc8_5UrnaqBm | 05f291f113624aefbf170f23a14f8604 | 10.2.3.184  | ERROR               | amphora  |
+--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+

(overcloud) [stack@undercloud ~]$ openstack loadbalancer delete c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 --debug
During handling of the above exception, another exception occurred:                                                                                                                                                
                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand                                                                                                                                
    result = cmd.run(parsed_args)                                                                                                                                                                                  
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/load_balancer.py", line 168, in take_action
    parsed_args)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 173, in get_loadbalancer_attrs                                                                                                      
    attrs = _map_attrs(_attrs, attr_map)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 35, in _map_attrs
    v,
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 118, in get_resource_id
    raise exceptions.CommandError(msg)
osc_lib.exceptions.CommandError: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers
clean_up DeleteLoadBalancer: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 115, in get_resource_id
    return names[0].get('id')
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/load_balancer.py", line 168, in take_action                                                                                                         
    parsed_args)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 173, in get_loadbalancer_attrs                                                                                                      
    attrs = _map_attrs(_attrs, attr_map)
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 35, in _map_attrs
    v,
  File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 118, in get_resource_id
    raise exceptions.CommandError(msg)
osc_lib.exceptions.CommandError: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers

END return value: 1

 

Version-Release number of selected component (if applicable):
RHOS-16.1-RHEL-8-20210205.n.0

How reproducible:
100%

Steps to Reproduce:
1.Deploy OSP with Octavia
2.Scale up to 2k+ LB's


Actual results:
Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers


Expected results:
Loadbalancer should be deleted successfully

Additional info:

The ``openStack loadbalancer list`` command only fetches 1k LB's as the pagination_max_limit = 1000 is set by default, so it's not able to delete LB's whose ID is not reported by ``openstack loadbalancer list`` even though we fetched the id of the LB that was moved to error appending ``--provisioning-status ERROR`` option

Comment 3 Korry Nguyen 2021-06-01 21:48:13 UTC

elevate to high pri/sev since it's listed as important by perf and scale team.

Comment 13 Asma Syed Hameed 2022-02-09 04:43:04 UTC

Bruna,
We are currently occupied with other Perf Scale activities. As it can be tested without Scale lab please use the procedure mentioned above.

Comment 22 errata-xmlrpc 2022-03-23 22:10:08 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1001