1718127 – Nova placement IndexError: list index out of range

Bug 1718127 - Nova placement IndexError: list index out of range

Summary: Nova placement IndexError: list index out of range

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	13.0 (Queens)
Hardware:	x86_64
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	z14
Target Release:	13.0 (Queens)
Assignee:	melanie witt
QA Contact:	OSP DFG:Compute
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-06-07 02:27 UTC by Sai Sindhur Malleni
Modified:	2023-12-15 16:32 UTC (History)
CC List:	18 users (show)
Fixed In Version:	openstack-nova-17.0.13-29.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-01-05 13:54:43 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
sos report (4.41 MB, application/x-xz) 2020-10-05 04:41 UTC, Itzik Brown	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1731668	None	None	None	2019-06-28 12:58:32 UTC
OpenStack gerrit	756636	None	MERGED	[stable-only] Add functional test for bug 1731668	2021-01-19 07:21:49 UTC
OpenStack gerrit	756637	None	MERGED	[stable-only] Use a separate transaction for reading after race	2021-01-19 07:21:49 UTC
Red Hat Product Errata	RHBA-2020:5578	None	None	None	2020-12-16 13:57:34 UTC

Description Sai Sindhur Malleni 2019-06-07 02:27:33 UTC

Description of problem:
On an OSP 13 cloud with 3 baremetal controllers+1 compute + 1000 simulated compute nodes (containers acting as compute nodes), we are trying to stress the cloud by running arally scenario that creates networks and launches VMs on that network 10000 times at a concurrency of 36. The VMs boot successfully, but we see the following error in Nova placement logs
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler [req-3fcd7185-8716-4fab-b5b6-8651ced22fa6 dea31cbab76e4a43887adf4ab5646c90 b1cbe0bc15a8409bbaf37d75828d7f4d - default default] Uncaught exception: IndexError: list index out of range
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler Traceback (most recent call last):
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handler.py", line 220, in __call__
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return dispatch(environ, start_response, self._map)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handler.py", line 147, in dispatch
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return handler(environ, start_response)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     resp = self.call_func(req, *args, **self.kwargs)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/wsgi_wrapper.py", line 29, in call_func
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     super(PlacementWsgify, self).call_func(req, *args, **kwargs)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return self.func(req, *args, **kwargs)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/microversion.py", line 261, in decorated_func
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return _find_method(f, version, status_code)(req, *args, **kwargs)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/util.py", line 167, in decorated_function
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return f(req)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handlers/allocation.py", line 305, in set_allocations_for_consumer
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return _set_allocations_for_consumer(req, schema.ALLOCATION_SCHEMA_V1_12)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/api/openstack/placement/handlers/allocation.py", line 264, in _set_allocations_for_consumer
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     allocations.create_all()
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/objects/resource_provider.py", line 2156, in create_all
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     self._set_allocations(self._context, self.objects)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     ectxt.value = e.inner_exc
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     self.force_reraise()
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     six.reraise(self.type_, self.value, self.tb)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return f(*args, **kwargs)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 988, in wrapper
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return fn(*args, **kwargs)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/objects/resource_provider.py", line 2072, in _set_allocations
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     alloc.ensure_consumer_project_user(context)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/objects/resource_provider.py", line 1703, in ensure_consumer_project_user
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     pid = _ensure_project(ctx, self.project_id)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/objects/resource_provider.py", line 1928, in _ensure_project
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return _ensure_lookup_table_entry(ctx, _PROJECT_TBL, external_id)
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler   File "/usr/lib/python2.7/site-packages/nova/objects/resource_provider.py", line 1916, in _ensure_lookup_table_entry
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler     return res[0][0]
2019-06-07 01:33:33.090 19 ERROR nova.api.openstack.placement.handler IndexError: list index out of range


Version-Release number of selected component (if applicable):
13

How reproducible:
100%

Steps to Reproduce:
1. deploy overcloud
2. Run rally scenarios that create networks and VMs (10000 Vms at a concurrency of 36)
3.

Actual results:
No errors in logs

Expected results:
index out of range error

Additional info:

Comment 3 Sai Sindhur Malleni 2019-06-21 17:47:46 UTC

I do not have sosreports. However, the reproducer is trying to boot a large number VMs parallely as stated in the initial comment.

Comment 7 Itzik Brown 2020-10-05 04:41:51 UTC

Created attachment 1718882 [details]
sos report

SOS report from my setup which had the same error puddle 2020-09-16.1

Comment 35 errata-xmlrpc 2020-12-16 13:57:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openstack-nova bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5578

Note You need to log in before you can comment on or make changes to this bug.