Bug 1215968

Reproduce it with latest rhel7.2 version:

kernel-3.10.0-246.el7.x86_64
qemu-kvm-rhev-2.3.0-1.el7.x86_64 
libvirt-1.2.14-1.el7.x86_64

How reproducible:
100%

Reproduce steps :

1.check USB device on host 
# lsusb
......
Bus 002 Device 004: ID 1005:b113 Apacer Technology, Inc. Handy Steno/AH123 / Handy Steno 2.0/HT203
......

2.start a healthy guest with following XML :

<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x1005'/>
<product id='0xb113'/>
</source>
</hostdev>

3.check domain XML , without libvirt-generated address .
# virsh dumpxml r7.2 | grep hostdev -A 9
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>    <== NO <address ...> element
    </hostdev>

4.start a guest without hostdev device .
# virsh dumpxml r7.2 | grep hostdev -A 9

5.attach a USB device and check domain XML 
without libvirt-generated address

# virsh attach-device r7.2 usb3.xml --persistent 
Device attached successfully

# virsh dumpxml r7.2 | grep hostdev -A 9
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>    <== NO <address ...> element
    </hostdev>

Expected results:
Should like PCI/SCSI device to add libvirt-generated address .

For PCI devices, libvirt needs to generate the addresses to preserve ABI compatibility across restarts/migration. AFAIK this is not be needed for USB devices.

Currently, for devices without a specified USB address libvirt lets QEMU choose the port and bus (and automatically add usb-hubs if needed).
Libvirt does not track those hubs and there does not seem to be a QMP command to retrieve the generated port numbers from QEMU.

If the user wants control over where the devices are plugged, or have control over the number/placement of USB hubs, this can be done by explicitly specifying the address:
<address type='usb' bus='0' port='1'/>
or adding a usb hub:
<hub type='usb'>
  <address type='usb' bus='0' port='2'/>
</hub>

What would be the use of the addresses? I am not sure why anyone would need to know where the device is, if they did not care to specify an address. (Also, this would require either copying the QEMU logic that chooses the ports, or a way to get the addresses from QEMU).

In general for all devices we don't care about the location initially when creating the VM. But on following starts(re-creation from libvirt POV) we need to preserve it. Hence we let qemu/libvirt to choose location initially and the retrieve the addresses once domain is started. And next "Run" sends the devices with addresses. 

getting address from qemu would be preferred

For a somewhat related issue - see bz 1176772

Hotplug of a device after initial configuration and then using managedsave/save and start/restore or a migration causes issues because we don't supply an address element.  In that bug if someone has a usb mouse or tablet, the restore/migration will fail with a fairly obtuse error message since qemu is managing the address space.

Like this case, the "workaround" is to provide the address for the tablet/mouse and then things work, but there's no guarantee of ordering other than the order libvirt uses for USB disk, chr, hostdev, and input devices.

*** Bug 1046267 has been marked as a duplicate of this bug. ***

First version of the patches proposed upstream:
https://www.redhat.com/archives/libvir-list/2015-August/msg00521.html

(In reply to Ján Tomko from comment #3)
> For PCI devices, libvirt needs to generate the addresses to preserve ABI
> compatibility across restarts/migration. AFAIK this is not be needed for USB
> devices.

It is needed for usb too, largely for the same reason it is needed for PCI.
The devices need to be connected using the same physical location (pci slot,
usb port) on source and target host.  Otherwise live migration will fail.

As long as you don't hotplug usb devices this isn't much of a problem in
practice.  But if you do you quickly run into cases where your usb devices
end up in different ports.

Easy reproducer: boot guest with two usb devices, hot-unplug the one which
comes first on the command line, then try migrate that guest.

> Currently, for devices without a specified USB address libvirt lets QEMU
> choose the port and bus (and automatically add usb-hubs if needed).
> Libvirt does not track those hubs and there does not seem to be a QMP
> command to retrieve the generated port numbers from QEMU.

You don't need to do that.  libvirt can simply assign usb ports,
just like it assigns pci slots.

uhci has two ports,
ohci has three ports.
ehci has six ports.
xhci has four ports (by default, is configurable).
hub has 8 ports.

> If the user wants control over where the devices are plugged, or have
> control over the number/placement of USB hubs, this can be done by
> explicitly specifying the address:
> <address type='usb' bus='0' port='1'/>
> or adding a usb hub:
> <hub type='usb'>
>   <address type='usb' bus='0' port='2'/>
> </hub>

Devices can be plugged into the hub using "port='2.${hub-port}'".

> What would be the use of the addresses? I am not sure why anyone would need
> to know where the device is, if they did not care to specify an address.

Choosing a port doesn't matter much, but keeping the port fixed matters.

> (Also, this would require either copying the QEMU logic that chooses the
> ports, or a way to get the addresses from QEMU).

No.

*** Bug 1211682 has been marked as a duplicate of this bug. ***

v2: https://www.redhat.com/archives/libvir-list/2016-June/msg01201.html

v3: https://www.redhat.com/archives/libvir-list/2016-June/msg01624.html

v4: https://www.redhat.com/archives/libvir-list/2016-July/msg00049.html
v5: https://www.redhat.com/archives/libvir-list/2016-July/msg00654.html

Pushed upstream as of:
commit bf182078d96476c8d5557737c1107241f280a947
Author:     Ján Tomko <jtomko>
CommitDate: 2016-07-21 08:30:26 +0200

    Assign addresses to USB devices
    
    Automatically assign addresses to USB devices.
    
    Just like reserving, this is only done for newly defined domains.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1215968

commit f2a781ceb075073a6033b96649f41501148d3c0c
Author:     Ján Tomko <jtomko>
CommitDate: 2016-07-21 08:30:26 +0200

    Assign addresses on USB device hotplug
    
    USB disks, redirected devices, host devices and serial devices
    are supported.

commit 815d98ac0bb8a4b48a412e026cb6465309e4043c
Author:     Ján Tomko <jtomko>
CommitDate: 2016-07-21 08:30:26 +0200

    Auto-add one hub if there are too many USB devices
    
    When parsing a command line with USB devices that have
    no address specified, QEMU automatically adds a USB hub
    if the device would fill up all the available USB ports.
    
    To help most of the users, add one hub if there are more
    USB devices than available ports. For wilder configurations,
    expect the user to provide us with more hubs and/or controllers.

git describe: v2.0.0-227-g815d98a

*** Bug 1176772 has been marked as a duplicate of this bug. ***

I'm curious about how your fix works while still letting old VMs migrate.

It does not assign addresses on migration.

USB addresses are only assigned for new domains:
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=8b04ce59

The newDomain parameter is true when:
* we parse a domain definition with the (internal) VIR_DOMAIN_DEF_PARSE_ABI_UPDATE flag set, that is defining a new domain
* we are preparing to start a new QEMU process (internal flag VIR_QEMU_PROCESS_START_NEW in qemuProcessPrepareDomain)

So if an old (running) VM was unable to migrate, it won't be able to migrate after this series either until it's restarted.

versions :
libvirt-2.0.0-8.el7.x86_64

Verify scenario1 : define start guest with all kinds of supportive USB devices, all the devices can generate address. 

Define a guest, as default it has USB companion controllers as following 

# virsh dumpxml vm2 | grep usb -A 5 
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>


Using virsh edit to add all the supportive usb devices into guests, add devices xml like following :
#virsh edit vm2 
......
<disk type='file' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/mnt/USB/usb.img'/>
  <target dev='sda' bus='usb'/>
</disk>
<controller type='ccid' index='0'/>
<smartcard mode='passthrough' type='spicevmc'>
      <address type='ccid' controller='0' slot='0'/>
</smartcard>
<input type='tablet' bus='usb'/>
<input type='keyboard' bus='usb'/>
<input type='mouse' bus='usb'/>
<hostdev mode='subsystem' type='usb'>
  <source startupPolicy='optional'>
    <vendor id='0x1005'/>
    <product id='0xb113'/>
  </source>
</hostdev>
<redirdev bus='usb' type='spicevmc'/>
......

save these configuration, and start guest
# virsh start vm2
Domain vm2 started

#virsh dumpxml vm2 | grep usb 
check all these USB devices can have an address. 

login guest to check
#lsusb -t 
can list all the usb devices
and using usb redirect can redirect an USB device into guest.

Verify scenario 2 : define start a guest with USB controller, hot plug USB devices, check hotpluged usb devices can generate address 

1.define and start a guest with usb controller, as default it has companion controller

2.Prepare a usb hostdev, attach successfully.
# cat hostdev1.xml 
<hostdev mode='subsystem' type='usb'>
  <source startupPolicy='optional'>
    <vendor id='0x1005'/>
    <product id='0xb113'/>
  </source>
</hostdev>

# virsh attach-device vm2 hostdev1.xml 
Device attached successfully

3.check it has generated address.

# virsh dumpxml vm2 | grep hostdev -A 5 
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

4.Prepare a virtual usb disk

# cat usb_disk.xml 
<disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdb'/>
      <target dev='sdb' bus='usb'/>
    </disk>

attach successfully and check it has generated address.

# virsh attach-device vm2 usb_disk.xml 
Device attached successfully

# virsh dumpxml vm2 | grep disk -A 5 
   <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdb'/>
      <backingStore/>
      <target dev='sdb' bus='usb'/>
      <alias name='usb-disk1'/>
      <address type='usb' bus='0' port='2'/>
    </disk>

scenario3 : hot plug an usb hostdev then do save restore and migration

1.define and start a guest with usb controller, as default it has companion controller

2. attach an USB hostdev device 

# virsh attach-device vm2 hostdev1.xml 
Device attached successfully

# virsh dumpxml vm2 | grep hostdev -A 5
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

3. check save and restore guest, it can save and restore successfully. 

# virsh save vm2 vm2.save

Domain vm2 saved to vm2.save

# virsh list 
 Id    Name                           State
----------------------------------------------------

# virsh restore vm2.save 
Domain restored from vm2.save

# virsh list 
 Id    Name                           State
----------------------------------------------------
 8     vm2                            running

# virsh dumpxml vm2 | grep hostdev -A 5
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

4. migrate from rhel7.3 to rhel7.3, migrate successfully.

# virsh migrate --live --verbose vm2 qemu+ssh://$IP/system --unsafe
root@$IP's password: 
Migration: [100 %]

on target :
# virsh dumpxml vm2  | grep hostdev -A 5
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional' missing='yes'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

migrate back to source :
# virsh migrate --live --verbose vm2 qemu+ssh://$IP/system --unsafe 
root@$IP's password: 
Migration: [100 %]

On src, check usb host device again
# virsh dumpxml vm2 | grep hostdev -A 5
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1005'/>
        <product id='0xb113'/>
        <address bus='2' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

scenario 4 : hot plug a virtual usb disk, then do save restore and migration

1. define and start a guest with usb controller, as default it has companion controller

# virsh list 
 Id    Name                           State
----------------------------------------------------
 17    vm2                            running

2.Prepae an usb disk, attach to guest successfully.

# cat usb_file_disk1.xml 
<disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/USB/usb1.img'/>
      <target dev='sdc' bus='usb'/>
    </disk>

# virsh attach-device vm2 usb_file_disk1.xml 
Device attached successfully

3. do save and restroe successfully 

# virsh save vm2 vm2.save 

Domain vm2 saved to vm2.save

# virsh restore vm2.save 
Domain restored from vm2.save

# virsh list 
 Id    Name                           State
----------------------------------------------------
 18    vm2                            running

# virsh dumpxml vm2|grep disk -A 9
 ......
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/USB/usb1.img'>
        <seclabel model='selinux' labelskip='yes'/>
      </source>
      <backingStore/>
      <target dev='sdc' bus='usb'/>
      <alias name='usb-disk2'/>
      <address type='usb' bus='0' port='1'/>
    </disk>

4. migrate from rhel7.3 to rhel7.3 

# virsh migrate --live --verbose vm2 qemu+ssh://$IP/system --unsafe
root@$IP's password: 
Migration: [100 %]

on the target, dumpxml to check :
# virsh dumpxml vm2|grep disk -A 9
   <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/USB/usb1.img'>
        <seclabel model='selinux' labelskip='yes'/>
      </source>
      <backingStore/>
      <target dev='sdc' bus='usb'/>
      <alias name='usb-disk2'/>
      <address type='usb' bus='0' port='1'/>
    </disk>

Migrate back to source :

# virsh migrate --live --verbose vm2 qemu+ssh://$IP/system --unsafe 
root@$IP's password: 
Migration: [100 %]

As above, move this bug to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html