Description of problem: Currently, we the Performance and Scale team are performing Octavia scale testing and we have scaled up to 2600 loadbalancers in our env using browbeat. Scenario used: https://github.com/cloud-bulldozer/browbeat/blob/master/rally/octavia/octavia-create-show-loadbalancers.yml MariaDB [octavia]> select COUNT(*) from load_balancer; +----------+ | COUNT(*) | +----------+ | 2600 | +----------+ 1 row in set (0.002 sec) MariaDB [octavia]> select COUNT(provisioning_status) from load_balancer where provisioning_status = "ACTIVE"; +----------------------------+ | COUNT(provisioning_status) | +----------------------------+ | 2599 | +----------------------------+ 1 row in set (0.002 sec) MariaDB [octavia]> select COUNT(provisioning_status) from load_balancer where provisioning_status = "ERROR"; +----------------------------+ | COUNT(provisioning_status) | +----------------------------+ | 1 | +----------------------------+ 1 row in set (0.000 sec) we are unable to delete the loadbalancer which was moved to the error state (overcloud) [stack@undercloud ~]$ openstack loadbalancer list --provisioning-status ERROR +--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+ | id | name | project_id | vip_address | provisioning_status | provider | +--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+ | c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 | s_rally_d694abc8_5UrnaqBm | 05f291f113624aefbf170f23a14f8604 | 10.2.3.184 | ERROR | amphora | +--------------------------------------+---------------------------+----------------------------------+-------------+---------------------+----------+ (overcloud) [stack@undercloud ~]$ openstack loadbalancer delete c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 --debug During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/load_balancer.py", line 168, in take_action parsed_args) File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 173, in get_loadbalancer_attrs attrs = _map_attrs(_attrs, attr_map) File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 35, in _map_attrs v, File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 118, in get_resource_id raise exceptions.CommandError(msg) osc_lib.exceptions.CommandError: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers clean_up DeleteLoadBalancer: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 115, in get_resource_id return names[0].get('id') IndexError: list index out of range During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run result = self.run_subcommand(remainder) File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/load_balancer.py", line 168, in take_action parsed_args) File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 173, in get_loadbalancer_attrs attrs = _map_attrs(_attrs, attr_map) File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 35, in _map_attrs v, File "/usr/lib/python3.6/site-packages/octaviaclient/osc/v2/utils.py", line 118, in get_resource_id raise exceptions.CommandError(msg) osc_lib.exceptions.CommandError: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers END return value: 1 Version-Release number of selected component (if applicable): RHOS-16.1-RHEL-8-20210205.n.0 How reproducible: 100% Steps to Reproduce: 1.Deploy OSP with Octavia 2.Scale up to 2k+ LB's Actual results: Unable to locate c6f3cfd6-3a92-4b86-af06-5b8afcb2a569 in loadbalancers Expected results: Loadbalancer should be deleted successfully Additional info: The ``openStack loadbalancer list`` command only fetches 1k LB's as the pagination_max_limit = 1000 is set by default, so it's not able to delete LB's whose ID is not reported by ``openstack loadbalancer list`` even though we fetched the id of the LB that was moved to error appending ``--provisioning-status ERROR`` option
elevate to high pri/sev since it's listed as important by perf and scale team.
Bruna, We are currently occupied with other Perf Scale activities. As it can be tested without Scale lab please use the procedure mentioned above.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.2), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1001