Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1583553

Summary: [OSP16][RFE] Not able to attach more than 26 virtio-scsi volumes
Product: Red Hat OpenStack Reporter: PURANDHAR SAIRAM MANNIDI <pmannidi>
Component: openstack-novaAssignee: melanie witt <mwitt>
Status: CLOSED DUPLICATE QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: aguetta, brault, broskos, cmuresan, dasmith, dwojewod, egallen, eglynn, jhakimra, kchamart, lyarwood, mbooth, mmahudha, mwitt, nagata3333333, pcbho, pmannidi, sbauza, sgordon, srevivo, stephenfin, vromanso
Target Milestone: Upstream M1Keywords: FutureFeature, Triaged
Target Release: 16.0 (Train on RHEL 8.1)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1635215 1635220 1691040 (view as bug list) Environment:
Last Closed: 2019-05-09 14:12:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1476902    
Bug Blocks: 1635215, 1635220, 1691040, 1691041    

Description PURANDHAR SAIRAM MANNIDI 2018-05-29 08:50:47 UTC 
Description of problem:
Not able to attach more than 26 virtio-scsi volumes
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions [req-4d22806c-f75f-4a16-8635-33aba2c7831b 245a7e95b00d476a83c21ec789873473 9003565472dc48b28826c53f3888f5aa - default default] Unexpected exception in API method
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/volumes.py", line 325, in create
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     volume_id, device)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 166, in inner
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 147, in inner
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3521, in attach_volume
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     disk_bus, device_type)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3458, in _attach_volume
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     device_type=device_type)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3440, in _create_volume_bdm
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     device_type=device_type)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 926, in reserve_block_device_name
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return cctxt.call(ctxt, 'reserve_block_device_name', **kw)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     retry=self.retry)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     timeout=timeout, retry=retry)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     retry=retry)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     raise result
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions NovaException_Remote: No free disk device names for prefix 'sd'
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     res = self.dispatcher.dispatch(message)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return self._do_dispatch(endpoint, method, ctxt, args)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     result = func(ctxt, **new_args)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     function_name, call_dict, binary)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return f(self, context, *args, **kw)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     LOG.warning(msg, e, instance=instance)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     kwargs['instance'], e, sys.exc_info())
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     self.force_reraise()
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     six.reraise(self.type_, self.value, self.tb)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return function(self, context, *args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4683, in reserve_block_device_name
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return do_reserve()
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4677, in do_reserve
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     instance, bdms, new_bdm)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1496, in _get_device_name_for_instance
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     instance, bdms, block_device_obj)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7962, in get_device_name_for_instance
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     block_device_obj, mapping=instance_info['mapping'])
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 394, in get_info_from_bdm
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     device_name = find_disk_dev_for_disk_bus(padded_mapping, bdm_bus)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 197, in find_disk_dev_for_disk_bus
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions     dev_prefix)
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions NovaException: No free disk device names for prefix 'sd'
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.076 14480 ERROR nova.api.openstack.extensions
2018-05-29 13:49:41.079 14480 INFO nova.api.openstack.wsgi [req-4d22806c-f75f-4a16-8635-33aba2c7831b 245a7e95b00d476a83c21ec789873473 9003565472dc48b28826c53f3888f5aa - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'NovaException_Remote'>
2018-05-29 13:49:41.079 14480 INFO nova.osapi_compute.wsgi.server [req-4d22806c-f75f-4a16-8635-33aba2c7831b 245a7e95b00d476a83c21ec789873473 9003565472dc48b28826c53f3888f5aa - default default] 10.74.128.53 "POST /v2.1/9003565472dc48b28826c53f3888f5aa/servers/d37043a1-f176-4b7c-8437-39c675b49dfc/os-volume_attachments HTTP/1.1" status: 500 len: 563 time: 0.2215559

Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform 10

How reproducible:
Always

Steps to Reproduce:
1. Create a instance with image having properties hw_scsi_model=virtio-scsi and hw_disk_bus=scsi
2. Try to attach 27 disks and nova fails after 26 disks are attached including the root disk.
3.

Actual results:
Nova fails after 26 disks

Expected results:
Nova should attach more than 100 disks

Additional info:
Comment 2 Sahid Ferdjaoui 2018-05-29 09:19:27 UTC 
We can't just increase the device limit for virtio-scsi. We also have do handle the device name that will be associated.

It seems that, in Nova we generate it with 'find_disk_dev_for_disk_bus'

    max_dev = get_dev_count_for_disk_bus(bus)
    devs = range(max_dev)

    for idx in devs:
	disk_dev = dev_prefix + chr(ord('a') + idx)
        if not has_disk_dev(mapping, disk_dev):
	    if disk_dev not in assigned_devices:
                return disk_dev

So from 0 to 25 we return a-z but I'm not sure what we should do then...

Daniel, you probably already have handled that with libvirt, what would be the solution?
Comment 3 PURANDHAR SAIRAM MANNIDI 2018-05-29 09:32:43 UTC 
we should return like aa-az,ba-bz,..etc
Comment 5 melanie witt 2018-05-31 16:47:48 UTC 
Recently, there is a patch proposed upstream to increase the number of volumes allowed to attach to a single instance > 26:

https://review.openstack.org/567472

but there's some debate about the new maximum we should select.
Comment 10 melanie witt 2018-06-07 22:49:04 UTC 
Thanks. I've started a thread upstream to get consensus in the community on an approach we can move forward with:

http://lists.openstack.org/pipermail/openstack-dev/2018-June/131289.html
Comment 18 melanie witt 2018-08-29 20:08:44 UTC 
I have proposed a spec upstream to make the maximum number of volumes attached to a single instance configurable:

https://review.openstack.org/597306
Comment 25 Stephen Finucane 2019-05-09 14:12:38 UTC 
Closing as duplicate of the OSP 15 bug

*** This bug has been marked as a duplicate of bug 1691040 ***