Bug 917702

Why usb-bot?  usb-storage automatically expands into two devices, a SCSI controller and a SCSI device.  This is a bad idea, and we've been struggling with its issues ever since.

usb-bot is a clean replacement for usb-storage.  The sooner we can deprecate usb-storage, the better.

As long as libvirt can't use usb-bot, we're stuck with usb-storage.  If we support usb-storage in 7.0, we're stuck with it throughout RHEL-7's lifetime.

(In reply to Markus Armbruster from comment #3)
> Why usb-bot?  usb-storage automatically expands into two devices, a SCSI
> controller and a SCSI device.  This is a bad idea, and we've been struggling
> with its issues ever since.
> 
> usb-bot is a clean replacement for usb-storage.  The sooner we can deprecate
> usb-storage, the better.
> 
> As long as libvirt can't use usb-bot, we're stuck with usb-storage.  If we
> support usb-storage in 7.0, we're stuck with it throughout RHEL-7's lifetime.

Hi Markus

I tried to add usb-uas support in libvirt before[1]. According to docs/usb-storage.txt, it said the qemu command line interface is simliar to usb-uas.
Could you give a tip what the difference between usb-uas and usb-bot is?

[1]https://www.redhat.com/archives/libvir-list/2013-January/msg00870.html

What we have today:
-------------------

<drive ...>
   <...>
   <target bus='usb'/>
</drive>

Result:
-drive if=none,id=${drive},...
-device usb-storage,drive=${drive},...


What we want have:
------------------

(1) The syntax above doing this instead if usb-bot is available:

-drive if=none,id=${drive},...
-device usb-bot,id=${bot},...
-device scsi-{hd,cd},bus=${bot}.0,drive=${drive},...

This is the most important one as we can avoid using the quirky usb-storage then.


(2) Support usb-bot (and usb-uas) as scsi controllers, i.e. syntax like this:

<controller type='scsi' index='42' model='usb-bot'/>
<drive ...>
  <...>
  <target bus='scsi'>
  <address type='drive' controller='42' bus='0' target='0' unit='0'/>
</drive>
<drive ...>
  <...>
  <target bus='scsi'>
  <address type='drive' controller='42' bus='0' target='0' unit='1'/>
</drive>

Should build this:

-drive if=none,id=${drive0},...
-drive if=none,id=${drive1},...
-device usb-bot,id=scsi42,...
-device scsi-{hd,cd},bus=scsi42.0,drive=${drive0},lun=0,...
-device scsi-{hd,cd},bus=scsi42.0,drive=${drive1},lun=1,...


From libvirt's point of view usb-bot and usb-uas are very simliar, so
it makes sense to simply do both while being at it.  They are scsi
controllers, supporting a single target and a number of luns.  For
usb-bot the limit is 15 luns, and they must be contigous (using lun 0
and 2 without lun 1 doesn't work).  For usb-uas the limit is 256, and
the luns can be sparse.


Little background:
------------------

usb-bot (bot == bulk only transport) is the classic protocol used by
usb sticks and usb cdrom drives and is universally supported.

usb-uas (uas == usb attached scsi) is a modern protocol designed for
usb 3.0.  Support for that is still emerging, in both hardware and
operating systems.

Let's compare usb-uas, usb-bot and usb-storage.

All three devices are SCSI HBAs.  They support only one SCSI target,
with SCSI ID 0.

From the guest's point of view, usb-bot and usb-storage are the exact
same device, while usb-uas is a different device.

The difference between usb-bot and usb-storage is how they're set up.

-device usb-storage creates itself, and additionally either a
scsi-disk or a scsi-generic device.  This is a hack.  No other -device
does such a thing.

scsi-disk is a legacy device that is either a hard disk or a CD-ROM,
depending on the media parameter of its drive.  New code should use
scsi-hd and scsi-cd instead, which have additional properties.

Example: creating usb-storage with a disk device
-drive if=none,file=foo.img,id=usb-drv0 \
-device usb-storage,id=usb-msd0,drive=usb-drv0

This creates a usb-storage device "usb-msd0", which provides SCSI bus
"usb-msd0.0", and a scsi-disk device (no ID, actually a hard disk) on
that bus.  info qtree:

          dev: usb-storage, id "usb-msd0"
            drive = <null>
            logical_block_size = 512
            physical_block_size = 512
            min_io_size = 0
            opt_io_size = 0
            bootindex = -1
            discard_granularity = 4294967295
            removable = off
            port = <null>
            serial = <null>
            full-path = on
            addr 0.0, port 1, speed 12, name QEMU USB MSD, attached
            bus: usb-msd0.0
              type SCSI
              dev: scsi-disk, id ""
                drive = usb-drv0
                logical_block_size = 512
                physical_block_size = 512
                min_io_size = 0
                opt_io_size = 0
                bootindex = -1
                discard_granularity = 4096
                ver = "1.6.50"
                serial = <null>
                vendor = "QEMU"
                product = "QEMU HARDDISK"
                removable = off
                dpofua = off
                wwn = 0x0
                channel = 0
                scsi-id = 0
                lun = 0

Note how usb-storage's properties have been copied to scsi-disk.  Its
drive property has been zapped, because only one device can use a
drive, and that must be scsi-disk.  Copying property serial means both
devices always have the same serial number, which is weird, but guests
don't mind.

usb-storage doesn't let you set scsi-disk properties ver, vendor,
product, dpofua, wwn, channel, scsi-id and lun.  scsi-id is not a
loss, because the HBA accepts only 0 anyway.

Connecting anything but a hard disk is pretty exotic.  USB sticks
commonly don't have tape drives inside :)

Example: creating usb-storage with a CD-ROM device
Command line as above, with media=cdrom added to -drive

Same HBA, but the scsi-disk device is now a CD-ROM:

              dev: scsi-disk, id ""
                drive = usb-drv0
                logical_block_size = 512
                physical_block_size = 512
                min_io_size = 0
                opt_io_size = 0
                bootindex = -1
                discard_granularity = 4096
                ver = "1.6.50"
                serial = <null>
                vendor = "QEMU"
                product = "QEMU CD-ROM"
                removable = on
                dpofua = off
                wwn = 0x0
                channel = 0
                scsi-id = 0
                lun = 0

Example: creating usb-storage with a SCSI generic device
Command line as above, except with file=/dev/sg1 (media doesn't matter):

Same HBA, but instead of scsi-disk we got scsi-generic:

              dev: scsi-generic, id ""
                drive = usb-drv0
                bootindex = -1
                channel = 0
                scsi-id = 0
                lun = 0

You see that usb-storage is excessively magic and somewhat unflexible.
That's why we hate it.

-device usb-bot creates just the SCSI HBA.  You create the SCSI device
connected to it with another -device.  This gives you full control
over the SCSI device's properties.

Example: creating usb-bot with a disk device

-drive if=none,file=tmp.qcow2,id=usb-drv0 \
-device usb-bot,id=usb-msd0 \
-device scsi-hd,id=usb-msd0-hd,bus=usb-msd0.0,drive=usb-drv0

This creates a usb-bot device "usb-msd0", which provides SCSI bus
"usb-msd0.0", and a scsi-hd device "usb-msd0-hd" on that bus.  info
qtree:

          dev: usb-bot, id "usb-msd0"
            port = <null>
            serial = <null>
            full-path = on
            addr 0.0, port 1, speed 12, name QEMU USB MSD, attached
            bus: usb-msd0.0
              type SCSI
              dev: scsi-hd, id "usb-msd0-hd"
                drive = usb-drv0
                logical_block_size = 512
                physical_block_size = 512
                min_io_size = 0
                opt_io_size = 0
                bootindex = -1
                discard_granularity = 4096
                ver = "1.6.50"
                serial = <null>
                vendor = "QEMU"
                product = "QEMU HARDDISK"
                removable = off
                dpofua = off
                wwn = 0x0
                cyls = 40
                heads = 16
                secs = 63
                channel = 0
                scsi-id = 0
                lun = 0

Note that I used scsi-hd instead of the legacy scsi-disk.

From the guest's point of view, this is exactly the same device as we
got above in the "usb-storage with a disk device" example.

Unlike above, you now have full control over scsi-hd's properties.

To replicate the other two usb-storage examples, use scsi-cd and
scsi-generic instead of scsi-hd.

To replace an existing usb-storage configuration by usb-bot, you morph

    -device usb-storage,id=ID,HBA-PROPS,COMMON-PROPS,TARGET-PROPS

into

    -device usb-bot,id=ID,HBA-PROPS,COMMON-PROPS \
    -device TARGET-DRIVER,COMMON-PROPS,TARGET-PROPS,bus=ID.0

where

    HBA-PROPS is port, full-path
    COMMON-PROPS is serial
    TARGET-PROPS is the rest

Since usb-bot gives you control over lun, you can add additional
devices.  This is somewhat exotic.  In my opinion, exposing this
feature in the libvirt API is *not* necessary to resolve this bug.
Can be done later.

Without this additional feature, *no* domain XML changes should be
necessary.

usb-uas is a different, more capable SCSI HBA.  It's set up just like
usb-bot.

If you have further questions, please don't hesistate to ask them
here, or contact me directly.

Thanks for this information
upstream patches sent out
https://www.redhat.com/archives/libvir-list/2013-September/msg00020.html

But this is only to add usb-bot controller. The next step is to replace usb-storage with usb-bot.

One question when I writing this patchset.
If there are disks with bus='scsi' using usb-bot already, that means there is at least one usb-bot in there, then, how to handle another disk with bus='usb'.
In the case of usb-storage, it is kind of easy, we just need to add one more -device usb-storage.
In the case of usb-bot, there are two choices. One is to add one more -device usb-bot. The other is to attach this disk to a existing usb-bot as a lun.
which one is better?

> One question when I writing this patchset.
> If there are disks with bus='scsi' using usb-bot already, that means there
> is at least one usb-bot in there, then, how to handle another disk with
> bus='usb'.
> In the case of usb-storage, it is kind of easy, we just need to add one more
> -device usb-storage.
> In the case of usb-bot, there are two choices. One is to add one more
> -device usb-bot. The other is to attach this disk to a existing usb-bot as a
> lun.
> which one is better?

Whatever the domain xml says ...

Each scsi controller has a 'index' field:

<controller type='scsi' index='7' ... >
                        ^^^^^^^^^

Each drive as a controller field in the address:

<disk ... >
  [ ... ]
  <address type='drive' controller='7' bus='0' target='0' unit='0'/>
                        ^^^^^^^^^^^^^^
</disk>

The controller field specifies which scsi controller the disk should be
attached to.  So the domain xml allows to model both cases and the user
can pick what he wants.

https://www.redhat.com/archives/libvir-list/2013-September/msg00130.html
As usb-bot hot-plug doesn't work at the moment as pointed out by Gerd, we can defer the support of the hot-plug/unplug feature from libvit POV and throw
an error to user.

Alternatively, just wait for the qemu support of usb-bot hot-plug/unplug.

I'd like to see usb-bot (+usb-uas) support added to libvirt, even if we can't support hotplug yet.

Which unfortunaly implies that replacing usb-storage with usb-bot doesn't work because usb-storage has hotplug support :(

The usb-bot hot-plug problem is an instance of the general composite
device hot-plug problem.

By "composite device" I mean a device consisting of multiple device
models that can only be plugged and unplugged as a whole.

We first encountered this with PCI multifunction devices, and hacked
around it ever since.  Time for a proper solution.  Upstream, of
course.

I figure the proper solution involves building the composite device
from its components before plugging it in, and keeping a notion of its
compositeness, so we can refuse unplug of components.

Guannan, I very much appreciate your investigation of usb-bot support
in libvirt.  We see clearer now.  Unfortunately, the clearer view is
one of yet another nut to crack, namely composite device hot plug.

I understand we can't replace usb-storage by usb-bot until it is
cracked.

Now my question: do you think libvirt will be able to replace
usb-storage by usb-bot seamlessly once we reach feature parity?  By
"seamlessly", I mean any XML that gives usb-storage devices now will
result in the equivalent usb-bot devices then.  Assume such equivalent
devices exist, as far as qemu-kvm is concerned.

(In reply to Markus Armbruster from comment #13)

> 
> Now my question: do you think libvirt will be able to replace
> usb-storage by usb-bot seamlessly once we reach feature parity?  By
> "seamlessly", I mean any XML that gives usb-storage devices now will
> result in the equivalent usb-bot devices then.  Assume such equivalent
> devices exist, as far as qemu-kvm is concerned.

Hi Markus,
Currently, libvirt only uses usb-storage controller in a disk XML like:
<disk ...>
   <...>
   <target bus='usb'/>
</disk>

and qemu and libvirt support hot-plug/unplug the disk as well as its usb-storage controller. In order to use usb-bot instead of usb-storage in this case, usb-bot needs to support host-plug/unplug. If we reach the feature parity, the seamless it can be.

usb-bot can also be used to a standalone scsi controller to which one or more disks can be attached. In libvirt, the xml is like

<controller type='scsi' index='0' model='usb-bot'/>

In this case, I think we can accepts the absence of hot-plug/unplug feature of usb-bot. And I will send v2 patches out soon to give an useful error when user want it.

If libvirt can switch from usb-storage to usb-bot without the guest noticing once the latter can be hot plugged and unplugged, and if we succeed in supporting only libvirt, then not having usb-bot support in 7.0 doesn't mean we're stuck with usb-storage for the life of RHEL-7.  Based on that, moving the bug back 7.1 sounds sensible to me.

Thanks for trying support usb-bot in 7.0!  We know more now.

usb-bot still isn't hotpluggable. Last qemu commig regarding this device states it explicitly:

commit e9fd12aa0dd734f1e3725d93a0afa8721fab3bd2
Author: Igor Mammedov <imammedo>
Date:   Fri Sep 26 09:28:35 2014 +0000

    usb-bot: Mark device as non hotpluggable
    
    usb-bot creates SCSI bus and immediately makes it
    non hotpluggable which was making not possible to
    hotplug usb-bot since QEMU would abort at
    bus_add_child(scsi-hd) time when usb-bot is
    realized.
    
    Mark usb-bot as not hotpluggable so that attempt
    to hotplug it would error out even before it gets
    to device initialization point.

Load balancing, Martine please have a look. Thanks.

So the usb-bot still looks not hot-pluggable.  Does anyone want this?  If yes, is there a corresponding BZ for QEMU about making it hot-pluggable?  If not, can we close this as WONTFIX for now?  Thanks for any info.

Yes, we still want usb-bot, but whether we can and want to get it in RHEL-7 is a fair question.

Gerd, have we made progress towards composite device hotplug upstream?  "I don't know" is a valid answer ;)

Not really.

For multifunction PCI we have some kind of hack (new in 2.5 IIRC):  If you plug function 0 last (and unplug it first) things will work because the pci layer now hides functions from the guest in case function 0 is not present.  

That doesn't help here though.  And I suspect now that pci is solved a generic composite device hotplug isn't going to magically emerge.

Maybe the easiest way forward would be to have a simliar hack in usb too.  The usb bus can also hide devices from the guest (see "attached" in "info qtree").  Happens for example with usb-host if you hot-plug the physical device.  We would only have to expose attach/detach ops for usb devices to management via monitor and also add an option to plug a usb-bot device in detached state.

What do you think?

Adding a special-purpose hack for USB is a possibility.  The PCI
hack's charm is that it's implicit --- the existing commands suffice.
I understand that for USB we'd need means to control attached/detached
state with the monitor.  Would that requir new USB-specific commands,
or could we make do with generic commands to control properties?

I agree a general solution is unlikely to magically emerge.  If we
want one, we need to push for it.  Whether that makes sense depends on
how much we need the capability, and on the cost of a general solution
vs. the cost of additional special-purpose hacks.

How could a general solution look like?  I can see two variations.

One, provide means to create devices disconnected.  To hot-plug a
composite device, create the container device unconnected, connect the
parts to the container, hot-plug the container.

Two, provide means to create devices connected, but without making
them visible to the guest just yet.  To hot-plig a composite device,
create and plug the container without making it visible to the guest,
connect the parts to the container, make the container visible.  This
is a generalization of the PCI hack.

> I understand that for USB we'd need means to control attached/detached
> state with the monitor.  Would that requir new USB-specific commands,
> or could we make do with generic commands to control properties?

Attached is a property already.  Still qdev IIRC.  Making that a qom
property and make it writeable should work.

> I agree a general solution is unlikely to magically emerge.  If we
> want one, we need to push for it.  Whether that makes sense depends on
> how much we need the capability, and on the cost of a general solution
> vs. the cost of additional special-purpose hacks.

I think we don't really have any use cases other than pci multifunction
and usb composite devices.  Maybe scsi (multi-lun targets), but that
can be solved simliar to pci (plugging lun #0 triggers the guest
visibility of the other ones).

> Two, provide means to create devices connected, but without making
> them visible to the guest just yet.  To hot-plug a composite device,
> create and plug the container without making it visible to the guest,
> connect the parts to the container, make the container visible.  This
> is a generalization of the PCI hack.

Using usb attach/detach would be this model too.

> Attached is a property already.

Well, it isn't, even though "info qtree" prints it.  But we can export it of course.

Looks surprisingly easy on a quick glance:
https://www.kraxel.org/cgit/qemu/log/?h=work/usb-bot-hotplug

(famous last words, totally untested so far ...)

What are the blockdev-add words to create a empty cdrom (i.e. what
"-drive if=none,id=foo,media=cdrom,readonly" gives me)?

> https://www.kraxel.org/cgit/qemu/log/?h=work/usb-bot-hotplug

One incremental fix pushed.  Seems to work fine.

Created attachment 1118023 [details]
test script

expects libvirt domain name as first argument.
path to test isos at start of script needs to be adjusted.

As far as I know, there is still no straightforward way to create an empty block backend with blockdev-add.  You'd have to resort to trickery like creating one with a suitable dummy medium, then remove it with x-blockdev-remove-medium.

There's still not enough information about how to properly do this in QEMU and I heard of no real demand for this.  Moving to upstream.

The device is now hotpluggable and it's preferred to the 'usb-storage' device.

commit b78ecd0998094a8b7e0f14c4888f3a6b525d14ff
Author: Gerd Hoffmann <kraxel>
Date:   Wed Jun 15 11:46:58 2016 +0200

    usb-bot: hotplug support

    This patch marks usb-bot as hot-pluggable device, makes attached
    property settable and turns off auto-attach in case the device
    was hotplugged.

    Hot-plugging a usb-bot device with one or more scsi devices can be
    done this way now:

      (1) device-add usb-bot,id=foo
      (2) device-add scsi-{hd,cd},bus=foo.0,lun=0
      (2b) optionally add more devices (luns 0 ... 15).
      (3) qom-set foo.attached = true

    Signed-off-by: Gerd Hoffmann <kraxel>
    Reviewed-by: Markus Armbruster <armbru>
    Message-id: 1465984019-28963-5-git-send-email-kraxel

There should not be any guest-visible difference for the single-lun version, so we might want to use it for disk bus='usb' as a replacement for usb-storage.

(In reply to Peter Krempa from comment #39)
> There should not be any guest-visible difference for the single-lun version,
> so we might want to use it for disk bus='usb' as a replacement for
> usb-storage.

What about migration data stream compatibility   ?  If that isn't compat then we can't switch from usb-storage to usb-bot. We would need to have explicit user user to request one over the other.

Why can't you switch to usb-bot for new machine types regardless of migration compatibility?

(In reply to Markus Armbruster from comment #41)
> Why can't you switch to usb-bot for new machine types regardless of
> migration compatibility?

The notion of tieing behaviour changes to "new" machine types only doesn't work outside of QEMU code. 

Regardless of the libvirt version in use, the "default" machine type hasn't changed in QEMU.

eg, if we have libvirt 5.0.0 and QEMU 3.1.0, the "new" machine type is "pc-3.1.0". If we then upgrade to libvirt 5.1.0, the "new" machine type is still "pc-3.1.0".

Libvirt 5.1.0 can't change the way it configures "pc-3.1.0" because that would create a migration incompatibility with how libvirt 5.0.0 used "pc-3.1.0".

> What about migration data stream compatibility   ?  

Same too.  Actually usb-storage + usb-bot are just a thin layer above a common, abstract base type.

(In reply to Gerd Hoffmann from comment #43)
> > What about migration data stream compatibility   ?  
> 
> Same too.  Actually usb-storage + usb-bot are just a thin layer above a
> common, abstract base type.

Good, that should be ok to switch then.

More generally though, proper usb-bot support requires more work in libvirt than just switching the command line we generate. docs/usb-storage.txt shows that it allows a single USB device to expose multiple LUNS. This should really be represented in libvirt by a <controller> and multiple <disks>. Currently we just have a single <disk> without any <controller>.  If we do this work, we can fairly easily also support the usb-uas device type.

(

For now, I'm dropping the following (cross-posted) upstream issue report from my queue, but I feel I should make a permanent record of it in this BZ.

[edk2] nonzero LUN on USB Bulk Only Transfer fails with QEMU+edk2
http://mid.mail-archive.com/cae4cdd7-2693-275b-5a54-e419e914d0a2@redhat.com
https://lists.01.org/pipermail/edk2-devel/2019-February/036633.html

[Qemu-devel] nonzero LUN on USB Bulk Only Transfer fails with QEMU+edk2
https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg03239.html

)

Thank you for reporting this issue to the libvirt project. Unfortunately we have been unable to resolve this issue due to insufficient maintainer capacity and it will now be closed. This is not a reflection on the possible validity of the issue, merely the lack of resources to investigate and address it, for which we apologise. If you none the less feel the issue is still important, you may choose to report it again at the new project issue tracker https://gitlab.com/libvirt/libvirt/-/issues The project also welcomes contribution from anyone who believes they can provide a solution.