Bug 1269577 - [RFE] Support more than six Virtio SCSI disks on a single bus controller
Summary: [RFE] Support more than six Virtio SCSI disks on a single bus controller
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 7.0 (Kilo)
Hardware: All
OS: Linux
medium
medium
Target Milestone: Upstream M1
: 13.0 (Queens)
Assignee: Sahid Ferdjaoui
QA Contact: Joe H. Rahme
URL:
Whiteboard:
Depends On:
Blocks: 1203710 1442136
TreeView+ depends on / blocked
 
Reported: 2015-10-07 15:12 UTC by Christian Horn
Modified: 2019-12-16 05:00 UTC (History)
26 users (show)

Fixed In Version: openstack-nova-17.0.0-0.20180103233857.0010e23.2.el7ost
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:26:22 UTC
Target Upstream Version:


Attachments (Terms of Use)
Script to create 10 SCSI disks to a single SCSI controller (1.11 KB, text/plain)
2016-06-22 15:08 UTC, Kashyap Chamarthy
no flags Details
Script to create 10 SCSI disks to a single SCSI controller (1.11 KB, application/x-shellscript)
2016-06-24 10:00 UTC, Kashyap Chamarthy
no flags Details


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 241063 'None' ABANDONED Improve nova attach volume api to support disk-bus 2020-09-11 06:19:39 UTC
Red Hat Bugzilla 1173144 None None None 2019-12-05 17:52:13 UTC
Red Hat Knowledge Base (Solution) 1758693 None None None Never
Red Hat Product Errata RHEA-2018:2086 normal SHIPPED_LIVE Red Hat OpenStack Platform 13.0 Enhancement Advisory 2018-06-28 19:51:39 UTC

Internal Links: 1173144

Description Christian Horn 2015-10-07 15:12:37 UTC
Description of problem:
Toplevel requirement:
Ability to assign disks which will be using virtio-scsi driver

details:
- customers use RHOS
- They can use "glance image-update --property hw_scsi_model=virtio-scsi <image>" to creates the virtio-scsi controller
- Then "glance image-update --property hw_disk_bus=scsi <image>" is ensuring that future assigned disks will be using scsi
- customers then start an instance (so really starting up a virtual guest)
- then customers assign additional disks.  Assigning >6 disks is leading to a new "storage controller" beeing assigned to the KVM guest, and it is not of type virtual guest virtio-scsi

Currently libvirt is not allowing to select that virtio-scsi should be used for the new disk.  Either blindly using the existing controller type for new controllers, or allowing the controllertype to be selected along with the disk would solve the issue.

Version-Release number of selected component (if applicable):
libvirt in all rhel and upstream

How reproducible:
always

Steps to Reproduce:
1. create virtio-scsi controller in a KVM guest
2. assign >6 new disks virtio-scsi

Actual results:
new disks are not of type virtio-scsi

Expected results:
new disks should be of type virtio-scsi


Additional info:
libvirt is not exposing this functionality on the lowest level.  After implementing this here, we need to work further ofcourse for the higher layers in RHOS.

Comment 1 Christian Horn 2015-10-07 15:15:37 UTC
Additional info:
- it seems that the most simple scsi standard limits to
  8 devices. 16 seems to be possible, but it's unclead how
  to configure it in libvirt (and we eventually need this
  exposed in openstack ofcourse)
- on https://access.redhat.com/articles/1436373 we
  state to support "Number of mounted volumes per host".
  How is this realized?
- A para-storage-driver which allows the required number
  of devices, and virtio-scsi comparable performance, might
  also fullfill the request

Comment 4 Ján Tomko 2015-11-12 16:07:09 UTC
The application attaching the disk can specify the address if the defaults chosen by libvirt are not good enough, e.g.:
  <address type='drive' controller='0' bus='0' target='0' unit='25'/>
This way more than 7 disks can be attached to a single virtio-scsi controller.
See http://www.redhat.com/archives/libvir-list/2013-November/msg01114.html

Auto-adding of controllers is a legacy "feature" and we should not be adding attributes to it. To have full control over the controller added, the application should add it separately.

I don't think there's anything more libvirt can do about this.

Comment 5 Christian Horn 2015-11-13 10:35:59 UTC
Is this available via virsh or only via libvirt?
I did not see a way to specify the unit-number in "virsh attach-disk".

Without specifying it (as also mentioned by you), the "unit" is just counted up to 7 and further disks are not attached.

Also when doing "virsh edit rhel7u1a" and adding following disk, the disk is not appearing in the started KVM guest:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel7u1a.qcow2'/>
      <target dev='sdz' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='25'/>
    </disk>

Disks with "sda" and "unit='1'" etc. are appearing.  Fedora23 as host.

Comment 6 Christian Horn 2015-11-13 10:56:35 UTC
Sorry for the noise, I understand from bz1173144 that attach-device is to be used.  Investigating if we have enough with this to make the functionality available in openstack, so the new disks can be attached from the outside.

From what I see, one to call
- with a custom scsi-device-identifier (sdb, sdc or such)
- and increase the the unit number
We will have to think over wether this housekeeping should be done in glance (transparent to whoever calls glance) or have to make the info available to the outside and call glance with these parameters.  Having it internally in libvirt would have taken off that burden, but lets see.

Comment 7 Christian Horn 2015-11-19 14:30:39 UTC
Hello openstack-glance team,

the overall goal of the request is in the description of this bz.
We have established that libvirt is providing the capability of attaching more than 8 disks, using "attach-device".  Can the functionality be made available via glance?

Comment 8 Flavio Percoco 2015-11-26 12:14:14 UTC
It's unclear to me what the exact request from a glance point of view is. Glance has no knowledge of the hypervisor in use and what nova can/cannot do with it. It's possible to assign properties to Glance but those are dynamic and they are consumed by Nova (or any other consumers of images).

Perhaps this should be moved to openstack-nova ?

Comment 9 Christian Horn 2015-12-15 10:32:23 UTC
Hi, could we get an update on this, please?

Comment 10 Kashyap Chamarthy 2015-12-21 12:17:39 UTC
(In reply to Christian Horn from comment #9)
> Hi, could we get an update on this, please?

Hi Christian,

From a libvirt perspective, this needs more testing with the below upstream libvirt fix (thanks to Daniel Berrangé for the pointer), which is part of the just-released (09DEC2015) libvirt-1.3.0.

   http://libvirt.org/git/?p=libvirt.git;a=commit;h=105794c -- 
   qemu: Automatic SCSI controller creation in SCSI disk hotplug broken

Comment 11 Christian Horn 2015-12-24 08:50:43 UTC
Thank you, Kashyap.
So using this patch, what would be the idea on libvirt level to approach this?
One would be able to always use the same command to attach devices, and libvirt would then if required create new scsi-type controllers by itself?

Can you give an example how to correctly trigger that attach operation using virsh?

With this I would then try the functionality (compiling 1.3 on fedora, or waiting until fedora updates to 1.3 codebase).  Would then need to request the code into RHEL7 (via a rebase or a port), and then consider if we need to change something in glance to trigger this via openstack API.

Comment 12 Christian Horn 2016-01-11 13:17:51 UTC
Hello, any updates to go forward here?

Comment 22 Kashyap Chamarthy 2016-06-22 14:57:38 UTC
Just to summarize: The current consensus from libvirt developers is that
to have more than six SCSI drives on a single bus, then one has to
manually specify those granular details (like 'unit' in the <address>
sub-element for each disk added).

Using 'attach-device' the below way -- which manually specifies the
'unit' attribute (and increments it for each disk) for the 'address'
sub-element (which describes where the device is placed on the virtual
bus that is presented to the guest) works if you want more than 6 disks
attached to a single controller.

I have attached a simple script (create-scsi-disks.bash) that creates 10
raw disk images and constructs a drive XML that attaches all the 10
disks to a single SCSI controller ("0").

Tested with:

    libvirt-daemon-kvm-1.3.5-2.fc24.x86_64
    qemu-system-x86-2.6.0-3.fc24.x86_64

And then you can see all the 11 disks attached to the *same* controller
(controller='0'):

  $ sudo virsh dumpxml cvm1 | egrep -i 'target|controller.*scsi' 
        <target dev='vda' bus='virtio'/>
        <target dev='sd' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='1'/>
        <target dev='sdb' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='2'/>
        <target dev='sdc' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='3'/>
        <target dev='sdd' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='4'/>
        <target dev='sde' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='5'/>
        <target dev='sdf' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='6'/>
        <target dev='sdg' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='7'/>
        <target dev='sdh' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='8'/>
        <target dev='sdi' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='9'/>
        <target dev='sdj' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='10'/>
        <target dev='sdk' bus='scsi'/>
        <address type='drive' controller='0' bus='0' target='0' unit='11'/>
  	  [...]

Comment 23 Kashyap Chamarthy 2016-06-22 15:08:53 UTC
Created attachment 1170826 [details]
Script to create 10 SCSI disks to a single SCSI controller

Comment 24 Kashyap Chamarthy 2016-06-24 10:00:49 UTC
Created attachment 1171901 [details]
Script to create 10 SCSI disks to a single SCSI controller

Comment 25 Stephen Gordon 2016-11-11 20:07:35 UTC
Kashyap, what are the next steps here for Nova?

Comment 27 Kashyap Chamarthy 2016-12-12 16:25:45 UTC
[Sigh, my browser crashed the other day, and my comment didn't make it through.]

Steve,

Will work with Vladik to resolve this.

And the current thinking on this is to find a way here, to store the SCSI controller ID in the instance; perhaps 'system_metadata' (in discussion with Vladik).  Once done, we could let more 'virti-scsi' devices to this specific controller.

Comment 28 Stephen Gordon 2016-12-20 21:51:49 UTC
(In reply to Kashyap Chamarthy from comment #27)
> [Sigh, my browser crashed the other day, and my comment didn't make it
> through.]
> 
> Steve,
> 
> Will work with Vladik to resolve this.
> 
> And the current thinking on this is to find a way here, to store the SCSI
> controller ID in the instance; perhaps 'system_metadata' (in discussion with
> Vladik).  Once done, we could let more 'virti-scsi' devices to this specific
> controller.

I'm assuming at this point that has to be a Pike specification?

Comment 29 Kashyap Chamarthy 2016-12-21 14:09:46 UTC
(In reply to Stephen Gordon from comment #28)
> (In reply to Kashyap Chamarthy from comment #27)
> > [Sigh, my browser crashed the other day, and my comment didn't make it
> > through.]
> > 
> > Steve,
> > 
> > Will work with Vladik to resolve this.
> > 
> > And the current thinking on this is to find a way here, to store the SCSI
> > controller ID in the instance; perhaps 'system_metadata' (in discussion with
> > Vladik).  Once done, we could let more 'virti-scsi' devices to this specific
> > controller.
> 
> I'm assuming at this point that has to be a Pike specification?

Yes, you're right.

Comment 33 Stephen Gordon 2017-01-24 21:27:13 UTC
Hi Kashyap,

Is there a draft specification up for this?

Thanks,

Steve

Comment 34 Sahid Ferdjaoui 2017-02-01 15:46:56 UTC
I probably missed something but I do not see a feature here, at least there is one to give capacity for users to select the controller used by a device but first at all we have a bug to fix.

After discussing with Vladik  we conclude that the root cause is in Nova, when using a virtio-iscsi controller we should specify the index of the controller which should be used by the device. A virtio-iscsi controller is able to handle 256 devices so for a guest using such iscsi model only one controller is necessary.

I think we should rename this issue to something like: "libvirt should allow up to 6 devices attached to a virtio-iscsi controller"

Comment 40 Stephen Gordon 2017-06-15 18:44:06 UTC
Moving to Queens, while the code may still land upstream in Pike it is not clear we will have time to complete verification.

Comment 43 Stephen Gordon 2017-07-04 13:57:05 UTC
Christian can you confirm which version of RHOSP is now in use at the relevant customer sites?

Comment 46 Christian Horn 2017-07-05 23:55:24 UTC
(In reply to Stephen Gordon from comment #43)
> Christian can you confirm which version of RHOSP is now in use at the
> relevant customer sites?

On base of #45, I think that no porting to releases should be done based on this bz, the functionality will eventually come into RHOS with a new major version.

Comment 47 Stephen Gordon 2017-07-28 18:46:23 UTC
Given the code for this landed in Pike, this would become TestOnly if verification falls to later releases.

Comment 48 Sahid Ferdjaoui 2017-09-25 07:50:30 UTC
The fix has been merged upstream for Pike and backported on stable/ocata [0].

I'm updating the target release to Pike and set the status to MODIFIED.

[0] https://review.openstack.org/#/q/status:merged+project:openstack/nova+topic:bug/1686116

Comment 57 Stephen Gordon 2018-06-01 01:50:22 UTC
For future time travelers, there was a check somewhere else that still limited all buses to 26 devices.

Refer to:

https://bugzilla.redhat.com/show_bug.cgi?id=1583553

Comment 59 errata-xmlrpc 2018-06-27 13:26:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.