Bug 1215968 - Libvirt does not generate guest USB addresses
Summary: Libvirt does not generate guest USB addresses
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: ---
Assignee: Ján Tomko
QA Contact: Virtualization Bugs
Yehuda Zimmerman
URL:
Whiteboard:
Keywords: FutureFeature
: 1046267 1176772 1211682 (view as bug list)
Depends On: 1211682
Blocks: 1305606 1003572
TreeView+ depends on / blocked
 
Reported: 2015-04-28 09:04 UTC by Michal Skrivanek
Modified: 2016-11-03 18:16 UTC (History)
19 users (show)

(edit)
The *libvirt* API generates addresses for USB devices

With this update, *libvirt* generates addresses for USB devices. These devices, along with the *libvirt*-generated address children can be found in the domain XML file. This ensures that future start, restore, and migrate operations have a consistent address for the guests' USB devices. As a result, you can migrate virtual machines to which USB devices have been attached.
Clone Of: 1211682
(edit)
Last Closed: 2016-11-03 18:16:15 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2577 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2016-11-03 12:07:06 UTC
Red Hat Bugzilla 1176772 None CLOSED Hot-plug a virtual usb device to guest, and then restore/migrate guest failed. 2019-05-16 13:30 UTC

Internal Trackers: 1176772

Description Michal Skrivanek 2015-04-28 09:04:31 UTC
+++ This bug was initially created as a clone of Bug #1211682 +++

Description of problem:
Libvirt does not generate addresses for USB host devices. For almost every other device (even PCI host devices), an address is generated and can be read via parsing domain XML.

Version-Release number of selected component (if applicable):
rpm -q libvirt
libvirt-1.2.13-1.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create a domain XML containing valid type="usb" hostdev
2. Start the domain

Actual results:
Domain XML contains hostdev element without additional address child.

Expected results:
Domain XML should contain hostdev element with libvirt-generated address child.

Additional info:

Comment 2 Pei Zhang 2015-05-04 03:32:46 UTC
Reproduce it with latest rhel7.2 version:

kernel-3.10.0-246.el7.x86_64
qemu-kvm-rhev-2.3.0-1.el7.x86_64 
libvirt-1.2.14-1.el7.x86_64

How reproducible:
100%

Reproduce steps :

1.check USB device on host 
# lsusb
......
Bus 002 Device 004: ID 1005:b113 Apacer Technology, Inc. Handy Steno/AH123 / Handy Steno 2.0/HT203
......

2.start a healthy guest with following XML :

<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x1005'/>
<product id='0xb113'/>
</source>
</hostdev>

3.check domain XML , without libvirt-generated address .
# virsh dumpxml r7.2 | grep hostdev -A 9
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>    <== NO <address ...> element
    </hostdev>

4.start a guest without hostdev device .
# virsh dumpxml r7.2 | grep hostdev -A 9

5.attach a USB device and check domain XML 
without libvirt-generated address

# virsh attach-device r7.2 usb3.xml --persistent 
Device attached successfully

# virsh dumpxml r7.2 | grep hostdev -A 9
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>    <== NO <address ...> element
    </hostdev>

Expected results:
Should like PCI/SCSI device to add libvirt-generated address .

Comment 3 Ján Tomko 2015-05-13 14:56:34 UTC
For PCI devices, libvirt needs to generate the addresses to preserve ABI compatibility across restarts/migration. AFAIK this is not be needed for USB devices.

Currently, for devices without a specified USB address libvirt lets QEMU choose the port and bus (and automatically add usb-hubs if needed).
Libvirt does not track those hubs and there does not seem to be a QMP command to retrieve the generated port numbers from QEMU.

If the user wants control over where the devices are plugged, or have control over the number/placement of USB hubs, this can be done by explicitly specifying the address:
<address type='usb' bus='0' port='1'/>
or adding a usb hub:
<hub type='usb'>
  <address type='usb' bus='0' port='2'/>
</hub>

What would be the use of the addresses? I am not sure why anyone would need to know where the device is, if they did not care to specify an address. (Also, this would require either copying the QEMU logic that chooses the ports, or a way to get the addresses from QEMU).

Comment 4 Michal Skrivanek 2015-05-14 06:52:37 UTC
In general for all devices we don't care about the location initially when creating the VM. But on following starts(re-creation from libvirt POV) we need to preserve it. Hence we let qemu/libvirt to choose location initially and the retrieve the addresses once domain is started. And next "Run" sends the devices with addresses. 

getting address from qemu would be preferred

Comment 5 John Ferlan 2015-06-05 12:22:04 UTC
For a somewhat related issue - see bz 1176772

Hotplug of a device after initial configuration and then using managedsave/save and start/restore or a migration causes issues because we don't supply an address element.  In that bug if someone has a usb mouse or tablet, the restore/migration will fail with a fairly obtuse error message since qemu is managing the address space.

Like this case, the "workaround" is to provide the address for the tablet/mouse and then things work, but there's no guarantee of ordering other than the order libvirt uses for USB disk, chr, hostdev, and input devices.

Comment 6 Martin Kletzander 2015-07-22 14:18:33 UTC
*** Bug 1046267 has been marked as a duplicate of this bug. ***

Comment 11 Ján Tomko 2015-08-12 14:55:47 UTC
First version of the patches proposed upstream:
https://www.redhat.com/archives/libvir-list/2015-August/msg00521.html

Comment 13 Gerd Hoffmann 2016-02-24 13:02:25 UTC
(In reply to Ján Tomko from comment #3)
> For PCI devices, libvirt needs to generate the addresses to preserve ABI
> compatibility across restarts/migration. AFAIK this is not be needed for USB
> devices.

It is needed for usb too, largely for the same reason it is needed for PCI.
The devices need to be connected using the same physical location (pci slot,
usb port) on source and target host.  Otherwise live migration will fail.

As long as you don't hotplug usb devices this isn't much of a problem in
practice.  But if you do you quickly run into cases where your usb devices
end up in different ports.

Easy reproducer: boot guest with two usb devices, hot-unplug the one which
comes first on the command line, then try migrate that guest.

> Currently, for devices without a specified USB address libvirt lets QEMU
> choose the port and bus (and automatically add usb-hubs if needed).
> Libvirt does not track those hubs and there does not seem to be a QMP
> command to retrieve the generated port numbers from QEMU.

You don't need to do that.  libvirt can simply assign usb ports,
just like it assigns pci slots.

uhci has two ports,
ohci has three ports.
ehci has six ports.
xhci has four ports (by default, is configurable).
hub has 8 ports.

> If the user wants control over where the devices are plugged, or have
> control over the number/placement of USB hubs, this can be done by
> explicitly specifying the address:
> <address type='usb' bus='0' port='1'/>
> or adding a usb hub:
> <hub type='usb'>
>   <address type='usb' bus='0' port='2'/>
> </hub>

Devices can be plugged into the hub using "port='2.${hub-port}'".

> What would be the use of the addresses? I am not sure why anyone would need
> to know where the device is, if they did not care to specify an address.

Choosing a port doesn't matter much, but keeping the port fixed matters.

> (Also, this would require either copying the QEMU logic that chooses the
> ports, or a way to get the addresses from QEMU).

No.

Comment 14 Cole Robinson 2016-04-10 21:04:03 UTC
*** Bug 1211682 has been marked as a duplicate of this bug. ***

Comment 19 Ján Tomko 2016-07-21 13:54:47 UTC
v4: https://www.redhat.com/archives/libvir-list/2016-July/msg00049.html
v5: https://www.redhat.com/archives/libvir-list/2016-July/msg00654.html

Pushed upstream as of:
commit bf182078d96476c8d5557737c1107241f280a947
Author:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2016-07-21 08:30:26 +0200

    Assign addresses to USB devices
    
    Automatically assign addresses to USB devices.
    
    Just like reserving, this is only done for newly defined domains.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1215968

commit f2a781ceb075073a6033b96649f41501148d3c0c
Author:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2016-07-21 08:30:26 +0200

    Assign addresses on USB device hotplug
    
    USB disks, redirected devices, host devices and serial devices
    are supported.

commit 815d98ac0bb8a4b48a412e026cb6465309e4043c
Author:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2016-07-21 08:30:26 +0200

    Auto-add one hub if there are too many USB devices
    
    When parsing a command line with USB devices that have
    no address specified, QEMU automatically adds a USB hub
    if the device would fill up all the available USB ports.
    
    To help most of the users, add one hub if there are more
    USB devices than available ports. For wilder configurations,
    expect the user to provide us with more hubs and/or controllers.

git describe: v2.0.0-227-g815d98a

Comment 22 John Ferlan 2016-07-25 18:09:10 UTC
*** Bug 1176772 has been marked as a duplicate of this bug. ***

Comment 23 Dr. David Alan Gilbert 2016-07-26 08:59:53 UTC
I'm curious about how your fix works while still letting old VMs migrate.

Comment 24 Ján Tomko 2016-07-26 10:05:35 UTC
It does not assign addresses on migration.

USB addresses are only assigned for new domains:
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=8b04ce59

The newDomain parameter is true when:
* we parse a domain definition with the (internal) VIR_DOMAIN_DEF_PARSE_ABI_UPDATE flag set, that is defining a new domain
* we are preparing to start a new QEMU process (internal flag VIR_QEMU_PROCESS_START_NEW in qemuProcessPrepareDomain)

So if an old (running) VM was unable to migrate, it won't be able to migrate after this series either until it's restarted.

Comment 27 Pei Zhang 2016-09-12 09:16:23 UTC
versions :
libvirt-2.0.0-8.el7.x86_64

Verify scenario1 : define start guest with all kinds of supportive USB devices, all the devices can generate address. 

Define a guest, as default it has USB companion controllers as following 

# virsh dumpxml vm2 | grep usb -A 5 
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>


Using virsh edit to add all the supportive usb devices into guests, add devices xml like following :
#virsh edit vm2 
......
<disk type='file' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/mnt/USB/usb.img'/>
  <target dev='sda' bus='usb'/>
</disk>
<controller type='ccid' index='0'/>
<smartcard mode='passthrough' type='spicevmc'>
      <address type='ccid' controller='0' slot='0'/>
</smartcard>
<input type='tablet' bus='usb'/>
<input type='keyboard' bus='usb'/>
<input type='mouse' bus='usb'/>
<hostdev mode='subsystem' type='usb'>
  <source startupPolicy='optional'>
    <vendor id='0x1005'/>
    <product id='0xb113'/>
  </source>
</hostdev>
<redirdev bus='usb' type='spicevmc'/>
......

save these configuration, and start guest
# virsh start vm2
Domain vm2 started

#virsh dumpxml vm2 | grep usb 
check all these USB devices can have an address. 

login guest to check
#lsusb -t 
can list all the usb devices
and using usb redirect can redirect an USB device into guest.

Comment 28 Pei Zhang 2016-09-12 09:18:43 UTC
Verify scenario 2 : define start a guest with USB controller, hot plug USB devices, check hotpluged usb devices can generate address 

1.define and start a guest with usb controller, as default it has companion controller

2.Prepare a usb hostdev, attach successfully.
# cat hostdev1.xml 
<hostdev mode='subsystem' type='usb'>
  <source startupPolicy='optional'>
    <vendor id='0x1005'/>
    <product id='0xb113'/>
  </source>
</hostdev>

# virsh attach-device vm2 hostdev1.xml 
Device attached successfully

3.check it has generated address.

# virsh dumpxml vm2 | grep hostdev -A 5 
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

4.Prepare a virtual usb disk

# cat usb_disk.xml 
<disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdb'/>
      <target dev='sdb' bus='usb'/>
    </disk>

attach successfully and check it has generated address.

# virsh attach-device vm2 usb_disk.xml 
Device attached successfully

# virsh dumpxml vm2 | grep disk -A 5 
   <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdb'/>
      <backingStore/>
      <target dev='sdb' bus='usb'/>
      <alias name='usb-disk1'/>
      <address type='usb' bus='0' port='2'/>
    </disk>

Comment 29 Pei Zhang 2016-09-12 09:33:06 UTC
scenario3 : hot plug an usb hostdev then do save restore and migration

1.define and start a guest with usb controller, as default it has companion controller

2. attach an USB hostdev device 

# virsh attach-device vm2 hostdev1.xml 
Device attached successfully

# virsh dumpxml vm2 | grep hostdev -A 5
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

3. check save and restore guest, it can save and restore successfully. 

# virsh save vm2 vm2.save

Domain vm2 saved to vm2.save

# virsh list 
 Id    Name                           State
----------------------------------------------------

# virsh restore vm2.save 
Domain restored from vm2.save

# virsh list 
 Id    Name                           State
----------------------------------------------------
 8     vm2                            running

# virsh dumpxml vm2 | grep hostdev -A 5
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

4. migrate from rhel7.3 to rhel7.3, migrate successfully.

# virsh migrate --live --verbose vm2 qemu+ssh://$IP/system --unsafe
root@$IP's password: 
Migration: [100 %]

on target :
# virsh dumpxml vm2  | grep hostdev -A 5
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional' missing='yes'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

migrate back to source :
# virsh migrate --live --verbose vm2 qemu+ssh://$IP/system --unsafe 
root@$IP's password: 
Migration: [100 %]

On src, check usb host device again
# virsh dumpxml vm2 | grep hostdev -A 5
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

Comment 30 Pei Zhang 2016-09-12 09:44:08 UTC
scenario 4 : hot plug a virtual usb disk, then do save restore and migration

1. define and start a guest with usb controller, as default it has companion controller

# virsh list 
 Id    Name                           State
----------------------------------------------------
 17    vm2                            running

2.Prepae an usb disk, attach to guest successfully.

# cat usb_file_disk1.xml 
<disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/USB/usb1.img'/>
      <target dev='sdc' bus='usb'/>
    </disk>

# virsh attach-device vm2 usb_file_disk1.xml 
Device attached successfully

3. do save and restroe successfully 

# virsh save vm2 vm2.save 

Domain vm2 saved to vm2.save

# virsh restore vm2.save 
Domain restored from vm2.save

# virsh list 
 Id    Name                           State
----------------------------------------------------
 18    vm2                            running

# virsh dumpxml vm2|grep disk -A 9
 ......
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/USB/usb1.img'>
        <seclabel model='selinux' labelskip='yes'/>
      </source>
      <backingStore/>
      <target dev='sdc' bus='usb'/>
      <alias name='usb-disk2'/>
      <address type='usb' bus='0' port='1'/>
    </disk>

4. migrate from rhel7.3 to rhel7.3 

# virsh migrate --live --verbose vm2 qemu+ssh://$IP/system --unsafe
root@$IP's password: 
Migration: [100 %]

on the target, dumpxml to check :
# virsh dumpxml vm2|grep disk -A 9
   <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/USB/usb1.img'>
        <seclabel model='selinux' labelskip='yes'/>
      </source>
      <backingStore/>
      <target dev='sdc' bus='usb'/>
      <alias name='usb-disk2'/>
      <address type='usb' bus='0' port='1'/>
    </disk>

Migrate back to source :

# virsh migrate --live --verbose vm2 qemu+ssh://$IP/system --unsafe 
root@$IP's password: 
Migration: [100 %]

Comment 33 Pei Zhang 2016-09-14 09:07:36 UTC
As above, move this bug to verified.

Comment 35 errata-xmlrpc 2016-11-03 18:16:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html


Note You need to log in before you can comment on or make changes to this bug.