Bug 917702 - [rfe] add usb-bot device support
[rfe] add usb-bot device support
Status: NEW
Product: Virtualization Tools
Classification: Community
Component: libvirt (Show other bugs)
unspecified
All Linux
unspecified Severity medium
: ---
: ---
Assigned To: Libvirt Maintainers
yisun
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-03-04 10:19 EST by Gunannan Ren
Modified: 2017-01-24 05:19 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 861345
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
test script (1.99 KB, text/plain)
2016-01-25 08:58 EST, Gerd Hoffmann
no flags Details

  None (edit)
Description Gunannan Ren 2013-03-04 10:19:00 EST
qemu 1.4 got a new usb device: usb-bot (basically usb-storage with a different configuration interface and multi-lun support).

See http://git.qemu.org/?p=qemu.git;a=blob;f=docs/usb-storage.txt, last paragraph.

Can you add support for that one too please?

--- Additional comment from Gunannan Ren on 2013-02-20 08:14:59 EST ---

(In reply to comment #10)
> qemu 1.4 got a new usb device: usb-bot (basically usb-storage with a
> different configuration interface and multi-lun support).
> 
> See http://git.qemu.org/?p=qemu.git;a=blob;f=docs/usb-storage.txt, last
> paragraph.
> 
> Can you add support for that one too please?

okay, I will do it.
Comment 3 Markus Armbruster 2013-08-12 11:19:37 EDT
Why usb-bot?  usb-storage automatically expands into two devices, a SCSI controller and a SCSI device.  This is a bad idea, and we've been struggling with its issues ever since.

usb-bot is a clean replacement for usb-storage.  The sooner we can deprecate usb-storage, the better.

As long as libvirt can't use usb-bot, we're stuck with usb-storage.  If we support usb-storage in 7.0, we're stuck with it throughout RHEL-7's lifetime.
Comment 5 Gunannan Ren 2013-08-23 02:52:59 EDT
(In reply to Markus Armbruster from comment #3)
> Why usb-bot?  usb-storage automatically expands into two devices, a SCSI
> controller and a SCSI device.  This is a bad idea, and we've been struggling
> with its issues ever since.
> 
> usb-bot is a clean replacement for usb-storage.  The sooner we can deprecate
> usb-storage, the better.
> 
> As long as libvirt can't use usb-bot, we're stuck with usb-storage.  If we
> support usb-storage in 7.0, we're stuck with it throughout RHEL-7's lifetime.

Hi Markus

I tried to add usb-uas support in libvirt before[1]. According to docs/usb-storage.txt, it said the qemu command line interface is simliar to usb-uas.
Could you give a tip what the difference between usb-uas and usb-bot is?

[1]https://www.redhat.com/archives/libvir-list/2013-January/msg00870.html
Comment 6 Gerd Hoffmann 2013-08-23 06:03:48 EDT
What we have today:
-------------------

<drive ...>
   <...>
   <target bus='usb'/>
</drive>

Result:
-drive if=none,id=${drive},...
-device usb-storage,drive=${drive},...


What we want have:
------------------

(1) The syntax above doing this instead if usb-bot is available:

-drive if=none,id=${drive},...
-device usb-bot,id=${bot},...
-device scsi-{hd,cd},bus=${bot}.0,drive=${drive},...

This is the most important one as we can avoid using the quirky usb-storage then.


(2) Support usb-bot (and usb-uas) as scsi controllers, i.e. syntax like this:

<controller type='scsi' index='42' model='usb-bot'/>
<drive ...>
  <...>
  <target bus='scsi'>
  <address type='drive' controller='42' bus='0' target='0' unit='0'/>
</drive>
<drive ...>
  <...>
  <target bus='scsi'>
  <address type='drive' controller='42' bus='0' target='0' unit='1'/>
</drive>

Should build this:

-drive if=none,id=${drive0},...
-drive if=none,id=${drive1},...
-device usb-bot,id=scsi42,...
-device scsi-{hd,cd},bus=scsi42.0,drive=${drive0},lun=0,...
-device scsi-{hd,cd},bus=scsi42.0,drive=${drive1},lun=1,...


From libvirt's point of view usb-bot and usb-uas are very simliar, so
it makes sense to simply do both while being at it.  They are scsi
controllers, supporting a single target and a number of luns.  For
usb-bot the limit is 15 luns, and they must be contigous (using lun 0
and 2 without lun 1 doesn't work).  For usb-uas the limit is 256, and
the luns can be sparse.


Little background:
------------------

usb-bot (bot == bulk only transport) is the classic protocol used by
usb sticks and usb cdrom drives and is universally supported.

usb-uas (uas == usb attached scsi) is a modern protocol designed for
usb 3.0.  Support for that is still emerging, in both hardware and
operating systems.
Comment 7 Markus Armbruster 2013-08-23 06:07:58 EDT
Let's compare usb-uas, usb-bot and usb-storage.

All three devices are SCSI HBAs.  They support only one SCSI target,
with SCSI ID 0.

From the guest's point of view, usb-bot and usb-storage are the exact
same device, while usb-uas is a different device.

The difference between usb-bot and usb-storage is how they're set up.

-device usb-storage creates itself, and additionally either a
scsi-disk or a scsi-generic device.  This is a hack.  No other -device
does such a thing.

scsi-disk is a legacy device that is either a hard disk or a CD-ROM,
depending on the media parameter of its drive.  New code should use
scsi-hd and scsi-cd instead, which have additional properties.

Example: creating usb-storage with a disk device
-drive if=none,file=foo.img,id=usb-drv0 \
-device usb-storage,id=usb-msd0,drive=usb-drv0

This creates a usb-storage device "usb-msd0", which provides SCSI bus
"usb-msd0.0", and a scsi-disk device (no ID, actually a hard disk) on
that bus.  info qtree:

          dev: usb-storage, id "usb-msd0"
            drive = <null>
            logical_block_size = 512
            physical_block_size = 512
            min_io_size = 0
            opt_io_size = 0
            bootindex = -1
            discard_granularity = 4294967295
            removable = off
            port = <null>
            serial = <null>
            full-path = on
            addr 0.0, port 1, speed 12, name QEMU USB MSD, attached
            bus: usb-msd0.0
              type SCSI
              dev: scsi-disk, id ""
                drive = usb-drv0
                logical_block_size = 512
                physical_block_size = 512
                min_io_size = 0
                opt_io_size = 0
                bootindex = -1
                discard_granularity = 4096
                ver = "1.6.50"
                serial = <null>
                vendor = "QEMU"
                product = "QEMU HARDDISK"
                removable = off
                dpofua = off
                wwn = 0x0
                channel = 0
                scsi-id = 0
                lun = 0

Note how usb-storage's properties have been copied to scsi-disk.  Its
drive property has been zapped, because only one device can use a
drive, and that must be scsi-disk.  Copying property serial means both
devices always have the same serial number, which is weird, but guests
don't mind.

usb-storage doesn't let you set scsi-disk properties ver, vendor,
product, dpofua, wwn, channel, scsi-id and lun.  scsi-id is not a
loss, because the HBA accepts only 0 anyway.

Connecting anything but a hard disk is pretty exotic.  USB sticks
commonly don't have tape drives inside :)

Example: creating usb-storage with a CD-ROM device
Command line as above, with media=cdrom added to -drive

Same HBA, but the scsi-disk device is now a CD-ROM:

              dev: scsi-disk, id ""
                drive = usb-drv0
                logical_block_size = 512
                physical_block_size = 512
                min_io_size = 0
                opt_io_size = 0
                bootindex = -1
                discard_granularity = 4096
                ver = "1.6.50"
                serial = <null>
                vendor = "QEMU"
                product = "QEMU CD-ROM"
                removable = on
                dpofua = off
                wwn = 0x0
                channel = 0
                scsi-id = 0
                lun = 0

Example: creating usb-storage with a SCSI generic device
Command line as above, except with file=/dev/sg1 (media doesn't matter):

Same HBA, but instead of scsi-disk we got scsi-generic:

              dev: scsi-generic, id ""
                drive = usb-drv0
                bootindex = -1
                channel = 0
                scsi-id = 0
                lun = 0

You see that usb-storage is excessively magic and somewhat unflexible.
That's why we hate it.

-device usb-bot creates just the SCSI HBA.  You create the SCSI device
connected to it with another -device.  This gives you full control
over the SCSI device's properties.

Example: creating usb-bot with a disk device

-drive if=none,file=tmp.qcow2,id=usb-drv0 \
-device usb-bot,id=usb-msd0 \
-device scsi-hd,id=usb-msd0-hd,bus=usb-msd0.0,drive=usb-drv0

This creates a usb-bot device "usb-msd0", which provides SCSI bus
"usb-msd0.0", and a scsi-hd device "usb-msd0-hd" on that bus.  info
qtree:

          dev: usb-bot, id "usb-msd0"
            port = <null>
            serial = <null>
            full-path = on
            addr 0.0, port 1, speed 12, name QEMU USB MSD, attached
            bus: usb-msd0.0
              type SCSI
              dev: scsi-hd, id "usb-msd0-hd"
                drive = usb-drv0
                logical_block_size = 512
                physical_block_size = 512
                min_io_size = 0
                opt_io_size = 0
                bootindex = -1
                discard_granularity = 4096
                ver = "1.6.50"
                serial = <null>
                vendor = "QEMU"
                product = "QEMU HARDDISK"
                removable = off
                dpofua = off
                wwn = 0x0
                cyls = 40
                heads = 16
                secs = 63
                channel = 0
                scsi-id = 0
                lun = 0

Note that I used scsi-hd instead of the legacy scsi-disk.

From the guest's point of view, this is exactly the same device as we
got above in the "usb-storage with a disk device" example.

Unlike above, you now have full control over scsi-hd's properties.

To replicate the other two usb-storage examples, use scsi-cd and
scsi-generic instead of scsi-hd.

To replace an existing usb-storage configuration by usb-bot, you morph

    -device usb-storage,id=ID,HBA-PROPS,COMMON-PROPS,TARGET-PROPS

into

    -device usb-bot,id=ID,HBA-PROPS,COMMON-PROPS \
    -device TARGET-DRIVER,COMMON-PROPS,TARGET-PROPS,bus=ID.0

where

    HBA-PROPS is port, full-path
    COMMON-PROPS is serial
    TARGET-PROPS is the rest

Since usb-bot gives you control over lun, you can add additional
devices.  This is somewhat exotic.  In my opinion, exposing this
feature in the libvirt API is *not* necessary to resolve this bug.
Can be done later.

Without this additional feature, *no* domain XML changes should be
necessary.

usb-uas is a different, more capable SCSI HBA.  It's set up just like
usb-bot.

If you have further questions, please don't hesistate to ask them
here, or contact me directly.
Comment 8 Gunannan Ren 2013-09-02 05:42:21 EDT
Thanks for this information
upstream patches sent out
https://www.redhat.com/archives/libvir-list/2013-September/msg00020.html

But this is only to add usb-bot controller. The next step is to replace usb-storage with usb-bot.

One question when I writing this patchset.
If there are disks with bus='scsi' using usb-bot already, that means there is at least one usb-bot in there, then, how to handle another disk with bus='usb'.
In the case of usb-storage, it is kind of easy, we just need to add one more -device usb-storage.
In the case of usb-bot, there are two choices. One is to add one more -device usb-bot. The other is to attach this disk to a existing usb-bot as a lun.
which one is better?
Comment 9 Gerd Hoffmann 2013-09-02 07:20:45 EDT
> One question when I writing this patchset.
> If there are disks with bus='scsi' using usb-bot already, that means there
> is at least one usb-bot in there, then, how to handle another disk with
> bus='usb'.
> In the case of usb-storage, it is kind of easy, we just need to add one more
> -device usb-storage.
> In the case of usb-bot, there are two choices. One is to add one more
> -device usb-bot. The other is to attach this disk to a existing usb-bot as a
> lun.
> which one is better?

Whatever the domain xml says ...

Each scsi controller has a 'index' field:

<controller type='scsi' index='7' ... >
                        ^^^^^^^^^

Each drive as a controller field in the address:

<disk ... >
  [ ... ]
  <address type='drive' controller='7' bus='0' target='0' unit='0'/>
                        ^^^^^^^^^^^^^^
</disk>

The controller field specifies which scsi controller the disk should be
attached to.  So the domain xml allows to model both cases and the user
can pick what he wants.
Comment 10 Gunannan Ren 2013-09-08 02:22:57 EDT
https://www.redhat.com/archives/libvir-list/2013-September/msg00130.html
As usb-bot hot-plug doesn't work at the moment as pointed out by Gerd, we can defer the support of the hot-plug/unplug feature from libvit POV and throw
an error to user.

Alternatively, just wait for the qemu support of usb-bot hot-plug/unplug.
Comment 11 Gerd Hoffmann 2013-09-09 02:05:09 EDT
I'd like to see usb-bot (+usb-uas) support added to libvirt, even if we can't support hotplug yet.

Which unfortunaly implies that replacing usb-storage with usb-bot doesn't work because usb-storage has hotplug support :(
Comment 12 Markus Armbruster 2013-09-10 03:27:40 EDT
The usb-bot hot-plug problem is an instance of the general composite
device hot-plug problem.

By "composite device" I mean a device consisting of multiple device
models that can only be plugged and unplugged as a whole.

We first encountered this with PCI multifunction devices, and hacked
around it ever since.  Time for a proper solution.  Upstream, of
course.

I figure the proper solution involves building the composite device
from its components before plugging it in, and keeping a notion of its
compositeness, so we can refuse unplug of components.
Comment 13 Markus Armbruster 2013-09-10 03:37:55 EDT
Guannan, I very much appreciate your investigation of usb-bot support
in libvirt.  We see clearer now.  Unfortunately, the clearer view is
one of yet another nut to crack, namely composite device hot plug.

I understand we can't replace usb-storage by usb-bot until it is
cracked.

Now my question: do you think libvirt will be able to replace
usb-storage by usb-bot seamlessly once we reach feature parity?  By
"seamlessly", I mean any XML that gives usb-storage devices now will
result in the equivalent usb-bot devices then.  Assume such equivalent
devices exist, as far as qemu-kvm is concerned.
Comment 14 Gunannan Ren 2013-09-10 04:11:19 EDT
(In reply to Markus Armbruster from comment #13)

> 
> Now my question: do you think libvirt will be able to replace
> usb-storage by usb-bot seamlessly once we reach feature parity?  By
> "seamlessly", I mean any XML that gives usb-storage devices now will
> result in the equivalent usb-bot devices then.  Assume such equivalent
> devices exist, as far as qemu-kvm is concerned.

Hi Markus,
Currently, libvirt only uses usb-storage controller in a disk XML like:
<disk ...>
   <...>
   <target bus='usb'/>
</disk>

and qemu and libvirt support hot-plug/unplug the disk as well as its usb-storage controller. In order to use usb-bot instead of usb-storage in this case, usb-bot needs to support host-plug/unplug. If we reach the feature parity, the seamless it can be.

usb-bot can also be used to a standalone scsi controller to which one or more disks can be attached. In libvirt, the xml is like

<controller type='scsi' index='0' model='usb-bot'/>

In this case, I think we can accepts the absence of hot-plug/unplug feature of usb-bot. And I will send v2 patches out soon to give an useful error when user want it.
Comment 16 Markus Armbruster 2013-09-18 10:36:31 EDT
If libvirt can switch from usb-storage to usb-bot without the guest noticing once the latter can be hot plugged and unplugged, and if we succeed in supporting only libvirt, then not having usb-bot support in 7.0 doesn't mean we're stuck with usb-storage for the life of RHEL-7.  Based on that, moving the bug back 7.1 sounds sensible to me.

Thanks for trying support usb-bot in 7.0!  We know more now.
Comment 22 Peter Krempa 2015-04-16 04:18:09 EDT
usb-bot still isn't hotpluggable. Last qemu commig regarding this device states it explicitly:

commit e9fd12aa0dd734f1e3725d93a0afa8721fab3bd2
Author: Igor Mammedov <imammedo@redhat.com>
Date:   Fri Sep 26 09:28:35 2014 +0000

    usb-bot: Mark device as non hotpluggable
    
    usb-bot creates SCSI bus and immediately makes it
    non hotpluggable which was making not possible to
    hotplug usb-bot since QEMU would abort at
    bus_add_child(scsi-hd) time when usb-bot is
    realized.
    
    Mark usb-bot as not hotpluggable so that attempt
    to hotplug it would error out even before it gets
    to device initialization point.
Comment 25 Jaroslav Suchanek 2016-01-12 08:56:07 EST
Load balancing, Martine please have a look. Thanks.
Comment 26 Martin Kletzander 2016-01-12 10:44:22 EST
So the usb-bot still looks not hot-pluggable.  Does anyone want this?  If yes, is there a corresponding BZ for QEMU about making it hot-pluggable?  If not, can we close this as WONTFIX for now?  Thanks for any info.
Comment 27 Markus Armbruster 2016-01-19 07:28:42 EST
Yes, we still want usb-bot, but whether we can and want to get it in RHEL-7 is a fair question.

Gerd, have we made progress towards composite device hotplug upstream?  "I don't know" is a valid answer ;)
Comment 28 Gerd Hoffmann 2016-01-19 08:18:03 EST
Not really.

For multifunction PCI we have some kind of hack (new in 2.5 IIRC):  If you plug function 0 last (and unplug it first) things will work because the pci layer now hides functions from the guest in case function 0 is not present.  

That doesn't help here though.  And I suspect now that pci is solved a generic composite device hotplug isn't going to magically emerge.

Maybe the easiest way forward would be to have a simliar hack in usb too.  The usb bus can also hide devices from the guest (see "attached" in "info qtree").  Happens for example with usb-host if you hot-plug the physical device.  We would only have to expose attach/detach ops for usb devices to management via monitor and also add an option to plug a usb-bot device in detached state.

What do you think?
Comment 29 Markus Armbruster 2016-01-25 05:06:24 EST
Adding a special-purpose hack for USB is a possibility.  The PCI
hack's charm is that it's implicit --- the existing commands suffice.
I understand that for USB we'd need means to control attached/detached
state with the monitor.  Would that requir new USB-specific commands,
or could we make do with generic commands to control properties?

I agree a general solution is unlikely to magically emerge.  If we
want one, we need to push for it.  Whether that makes sense depends on
how much we need the capability, and on the cost of a general solution
vs. the cost of additional special-purpose hacks.

How could a general solution look like?  I can see two variations.

One, provide means to create devices disconnected.  To hot-plug a
composite device, create the container device unconnected, connect the
parts to the container, hot-plug the container.

Two, provide means to create devices connected, but without making
them visible to the guest just yet.  To hot-plig a composite device,
create and plug the container without making it visible to the guest,
connect the parts to the container, make the container visible.  This
is a generalization of the PCI hack.
Comment 30 Gerd Hoffmann 2016-01-25 05:54:22 EST
> I understand that for USB we'd need means to control attached/detached
> state with the monitor.  Would that requir new USB-specific commands,
> or could we make do with generic commands to control properties?

Attached is a property already.  Still qdev IIRC.  Making that a qom
property and make it writeable should work.

> I agree a general solution is unlikely to magically emerge.  If we
> want one, we need to push for it.  Whether that makes sense depends on
> how much we need the capability, and on the cost of a general solution
> vs. the cost of additional special-purpose hacks.

I think we don't really have any use cases other than pci multifunction
and usb composite devices.  Maybe scsi (multi-lun targets), but that
can be solved simliar to pci (plugging lun #0 triggers the guest
visibility of the other ones).

> Two, provide means to create devices connected, but without making
> them visible to the guest just yet.  To hot-plug a composite device,
> create and plug the container without making it visible to the guest,
> connect the parts to the container, make the container visible.  This
> is a generalization of the PCI hack.

Using usb attach/detach would be this model too.
Comment 31 Gerd Hoffmann 2016-01-25 06:17:42 EST
> Attached is a property already.

Well, it isn't, even though "info qtree" prints it.  But we can export it of course.
Comment 32 Gerd Hoffmann 2016-01-25 07:08:48 EST
Looks surprisingly easy on a quick glance:
https://www.kraxel.org/cgit/qemu/log/?h=work/usb-bot-hotplug

(famous last words, totally untested so far ...)
Comment 33 Gerd Hoffmann 2016-01-25 07:52:22 EST
What are the blockdev-add words to create a empty cdrom (i.e. what
"-drive if=none,id=foo,media=cdrom,readonly" gives me)?
Comment 34 Gerd Hoffmann 2016-01-25 08:56:44 EST
> https://www.kraxel.org/cgit/qemu/log/?h=work/usb-bot-hotplug

One incremental fix pushed.  Seems to work fine.
Comment 35 Gerd Hoffmann 2016-01-25 08:58 EST
Created attachment 1118023 [details]
test script

expects libvirt domain name as first argument.
path to test isos at start of script needs to be adjusted.
Comment 36 Markus Armbruster 2016-01-29 09:19:35 EST
As far as I know, there is still no straightforward way to create an empty block backend with blockdev-add.  You'd have to resort to trickery like creating one with a suitable dummy medium, then remove it with x-blockdev-remove-medium.
Comment 38 Martin Kletzander 2017-01-24 05:19:35 EST
There's still not enough information about how to properly do this in QEMU and I heard of no real demand for this.  Moving to upstream.

Note You need to log in before you can comment on or make changes to this bug.