1583553 – [OSP16][RFE] Not able to attach more than 26 virtio-scsi volumes

Bug 1583553 - [OSP16][RFE] Not able to attach more than 26 virtio-scsi volumes

Summary: [OSP16][RFE] Not able to attach more than 26 virtio-scsi volumes

Keywords:
Status:	CLOSED DUPLICATE of bug 1691040
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	Upstream M1
Target Release:	16.0 (Train on RHEL 8.1)
Assignee:	melanie witt
QA Contact:	OSP DFG:Compute
Docs Contact:
URL:
Whiteboard:
Depends On:	1476902
Blocks:	1635215 1635220 1691040 1691041
TreeView+	depends on / blocked

Reported:	2018-05-29 08:50 UTC by PURANDHAR SAIRAM MANNIDI
Modified:	2023-03-21 18:51 UTC (History)
CC List:	22 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1635215 1635220 1691040 (view as bug list)
Environment:
Last Closed:	2019-05-09 14:12:38 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1773941	None	None	None	2018-05-29 09:19:14 UTC
OpenStack gerrit	616777	'None'	MERGED	Add configuration of maximum disk devices to attach	2021-01-16 22:10:35 UTC
OpenStack gerrit	632904	'None'	MERGED	Raise 403 instead of 500 error from attach volume API	2021-01-16 22:10:36 UTC
Red Hat Issue Tracker	OSP-11416	None	None	None	2021-12-10 16:26:03 UTC

Description PURANDHAR SAIRAM MANNIDI 2018-05-29 08:50:47 UTC

Description of problem:
Not able to attach more than 26 virtio-scsi volumes
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions [req-4d22806c-f75f-4a16-8635-33aba2c7831b 245a7e95b00d476a83c21ec789873473 9003565472dc48b28826c53f3888f5aa - default default] Unexpected exception in API method
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/volumes.py", line 325, in create
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     volume_id, device)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3521, in attach_volume
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     disk_bus, device_type)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3458, in _attach_volume
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     device_type=device_type)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3440, in _create_volume_bdm
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     device_type=device_type)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 926, in reserve_block_device_name
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     retry=self.retry)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     retry=retry)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     raise result
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions NovaException_Remote: No free disk device names for prefix 'sd'
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     res = self.dispatcher.dispatch(message)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return self._do_dispatch(endpoint, method, ctxt, args)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     result = func(ctxt, **new_args)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     function_name, call_dict, binary)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return f(self, context, *args, **kw)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     LOG.warning(msg, e, instance=instance)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     kwargs['instance'], e, sys.exc_info())
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4683, in reserve_block_device_name
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return do_reserve()
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4677, in do_reserve
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     instance, bdms, new_bdm)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1496, in _get_device_name_for_instance
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     instance, bdms, block_device_obj)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7962, in get_device_name_for_instance
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     block_device_obj, mapping=instance_info['mapping'])
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 394, in get_info_from_bdm
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 197, in find_disk_dev_for_disk_bus
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     dev_prefix)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions NovaException: No free disk device names for prefix 'sd'
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.079 14480 INFO nova.api.openstack.wsgi [req-4d22806c-f75f-4a16-8635-33aba2c7831b 245a7e95b00d476a83c21ec789873473 9003565472dc48b28826c53f3888f5aa - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'NovaException_Remote'>
2018-05-29 13:49:41.079 14480 INFO nova.osapi_compute.wsgi.server [req-4d22806c-f75f-4a16-8635-33aba2c7831b 245a7e95b00d476a83c21ec789873473 9003565472dc48b28826c53f3888f5aa - default default] 10.74.128.53 "POST /v2.1/9003565472dc48b28826c53f3888f5aa/servers/d37043a1-f176-4b7c-8437-39c675b49dfc/os-volume_attachments HTTP/1.1" status: 500 len: 563 time: 0.2215559

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform 10

How reproducible:
Always

Steps to Reproduce:
1. Create a instance with image having properties hw_scsi_model=virtio-scsi and hw_disk_bus=scsi
2. Try to attach 27 disks and nova fails after 26 disks are attached including the root disk.
3.

Actual results:
Nova fails after 26 disks

Expected results:
Nova should attach more than 100 disks

Additional info:

Comment 2 Sahid Ferdjaoui 2018-05-29 09:19:27 UTC

We can't just increase the device limit for virtio-scsi. We also have do handle the device name that will be associated.

It seems that, in Nova we generate it with 'find_disk_dev_for_disk_bus'

    max_dev = get_dev_count_for_disk_bus(bus)
    devs = range(max_dev)

    for idx in devs:
	disk_dev = dev_prefix + chr(ord('a') + idx)
        if not has_disk_dev(mapping, disk_dev):
	    if disk_dev not in assigned_devices:
                return disk_dev

So from 0 to 25 we return a-z but I'm not sure what we should do then...

Daniel, you probably already have handled that with libvirt, what would be the solution?

Comment 3 PURANDHAR SAIRAM MANNIDI 2018-05-29 09:32:43 UTC

we should return like aa-az,ba-bz,..etc

Comment 5 melanie witt 2018-05-31 16:47:48 UTC

Recently, there is a patch proposed upstream to increase the number of volumes allowed to attach to a single instance > 26:

https://review.openstack.org/567472

but there's some debate about the new maximum we should select.

Comment 10 melanie witt 2018-06-07 22:49:04 UTC

Thanks. I've started a thread upstream to get consensus in the community on an approach we can move forward with:

http://lists.openstack.org/pipermail/openstack-dev/2018-June/131289.html

Comment 18 melanie witt 2018-08-29 20:08:44 UTC

I have proposed a spec upstream to make the maximum number of volumes attached to a single instance configurable:

https://review.openstack.org/597306

Comment 25 Stephen Finucane 2019-05-09 14:12:38 UTC

Closing as duplicate of the OSP 15 bug

*** This bug has been marked as a duplicate of bug 1691040 ***

Note You need to log in before you can comment on or make changes to this bug.

aguetta
brault
broskos
cmuresan
dasmith
dwojewod
egallen
eglynn
jhakimra
kchamart
lyarwood
mbooth
mmahudha
mwitt
nagata3333333
pcbho
pmannidi
sbauza
sgordon
srevivo
stephenfin
vromanso