Bug 843560

Jirka, what's going on with this BZ?

At least from the spice usb-redir migration pov, after adding seamless migration support, no additional changes are needed for USB migration, so I believe that this bug can be closed.

qemuMigrationIsAllowed is not called during save/restore, which means
everything will work just fine when saving/restoring a domain with USB passed
through devices as long as the devices remain attach at the same address. In
other words, if the host is not rebooted and the device is not detached from
the host in the meantime. To make it work even if any of this happens, we need
to make some changes to libvirt.

First, we may add support for startupPolicy to USB devices which (similarly to
disk devices) would tell if the device is really required to be present when
starting or migrating a domain. With this, we could support device unplug
between save and restore since qemu should support -device usb-host,id=...
with no address and thus no real USB device.

To support change in USB address, we would need to (partially) ignore the USB
address in domain XML (since libvirt is able to find the right device using
vendor/product), however, this would not work that well in case several
identical devices are attached to the host and we need to make sure we use the
right one. I'll try to think more about how to handle this.

Both issues are fixed with the series I sent upstream yesterday: https://www.redhat.com/archives/libvir-list/2012-October/msg00335.html

This is now implemented upstream by commits v0.10.2-129-ge658dae through v0.10.2-134-gbd1282d and v0.10.2-139-gf95560b:

commit e658daeb58ec88deec5dd60fd3279f4feb4eac5b
Author: Jiri Denemark <jdenemar>
Date:   Tue Oct 2 15:14:02 2012 +0200

    conf: Add support for startupPolicy for USB devices
    
    USB devices can disappear without OS being mad about it, which makes
    them ideal for startupPolicy. With this attribute, USB devices can be
    configured to be mandatory (the default), requisite (will disappear
    during migration if they cannot be found), or completely optional.

commit 7bcc7278bf145e2628831823f77192307f09bfa1
Author: Jiri Denemark <jdenemar>
Date:   Wed Oct 3 13:55:35 2012 +0200

    qemu: Introduce qemuFindHostdevUSBDevice
    
    The code which looks up a USB device specified by hostdev is duplicated
    in two places. This patch creates a dedicated function that can be
    called in both places.

commit 059aff6b9860b18634709bc76d64ea78f7e609db
Author: Jiri Denemark <jdenemar>
Date:   Wed Oct 3 16:57:28 2012 +0200

    qemu: Add option to treat missing USB devices as success
    
    All USB device lookup functions emit an error when they cannot find the
    requested device. With this patch, their caller can choose if a missing
    device is an error or normal condition.

commit edc9269a2a80082a6898e14b1c10b4a94197b731
Author: Jiri Denemark <jdenemar>
Date:   Thu Oct 4 16:18:16 2012 +0200

    qemu: Implement startupPolicy for USB passed through devices

commit 28f8dfdcccd4c0f69063ef741545b37d8a7f7935
Author: Jiri Denemark <jdenemar>
Date:   Mon Oct 8 11:58:05 2012 +0200

    Add MIGRATABLE flag for virDomainGetXMLDesc
    
    Using VIR_DOMAIN_XML_MIGRATABLE flag, one can request domain's XML
    configuration that is suitable for migration or save/restore. Such XML
    may contain extra run-time stuff internal to libvirt and some default
    configuration may be removed for better compatibility of the XML with
    older libvirt releases.
    
    This flag may serve as an easy way to get the XML that can be passed
    (after desired modifications) to APIs that accept custom XMLs, such as
    virDomainMigrate{,ToURI}2 or virDomainSaveFlags.

commit bd1282d6247742d732f0076811521cfdc6b13472
Author: Jiri Denemark <jdenemar>
Date:   Tue Oct 9 13:15:46 2012 +0200

    qemu: Make save/restore with USB devices usable
    
    Save/restore with passed through USB devices currently only works if the
    USB device can be found at the same USB address where it used to be
    before saving a domain. This makes sense in case a user explicitly
    configure the USB address in domain XML. However, if the device was
    found automatically by vendor/product identification, we should try to
    search for that device when restoring the domain and use any device we
    find as long as there is only one available. In other words, the USB
    device can now be removed and plugged again or the host can be rebooted
    between saving and restoring the domain.

commit f95560b3fe5e239ed85ec01885ed5eb872439e98
Author: Jiri Denemark <jdenemar>
Date:   Thu Oct 11 22:33:46 2012 +0200

    conf: Mark missing optional USB devices in domain XML
    
    When startupPolicy set for a USB devices allows such device to be
    missing, there was no way this could be detected from domain XML. With
    this patch, libvirt emits a new missing='yes' attribute for such devices
    when active domain XML is generated.

In POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2012-October/msg00601.html

Test on 
qemu-kvm-0.12.1.2-2.327.el6.x86_64
libvirt-0.10.2-3.el6.x86_64
kernel-2.6.32-329.el6.x86_64

Migrate a guest with
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source startupPolicy='requisite'>
        <address bus='2' device='2'/>
      </source>
      <alias name='hostdev0'/>
    </hostdev>

report error
error: Requested operation is not valid: Domain with assigned host devices cannot be migrated

and with --migratable, it ony remove the 
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
device, but if I set 
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <address bus='2' device='2'/>
      </source>
      <alias name='hostdev0'/>
    </hostdev>
there is no change in dumpxml --migratable, is that right?

Oops, I forgot to change one check that forbids migration ir there are any hostdev devices configured for the domain. It will work for save/restore/managedsave scenario, though. I'll prepare the patch for fixing the migration check and include it in the next build.

BTW, to really test all scenarios (such as usb device replug after managedsaving a domain), you should rather identify the usb device using vendor/product. And the --migratable flag will have greater impact in that case, too. It should remove the automatically detected address of the device.

Actually, the automatically detected address should still be there but a new attribute autoAddress='yes' should appeare in the source element. I'm sorry about the confusion.

The missing part is fixed upstream by v0.10.2-204-g9fcc543:

commit 9fcc5436d3d824ceb82b44852e1a8023c3810655
Author: Jiri Denemark <jdenemar>
Date:   Fri Oct 19 13:36:33 2012 +0200

    qemu: Allow migration with host USB devices
    
    Recently, patches were added support for (managed)saving, restoring, and
    migrating domains with host USB devices. However, qemu driver would
    still forbid migration of such domains because qemuMigrationIsAllowed
    was not updated.

Back in POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2012-October/msg00998.html

Verify pass on
qemu-kvm-rhev-0.12.1.2-2.317.el6.x86_64
libvirt-0.10.2-5.el6.x86_64
kernel-2.6.32-329.el6.x86_64

Test scenario 
1. With startupPolicy='requisite'
virsh start vr-rhel6u3-x86_64-kvm
error: Failed to start domain vr-rhel6u3-x86_64-kvm
error: internal error Did not find USB device 951:1646

Migration, restore succeed

2. With startupPolicy='optional'
Start, Migration and restore succeed

3. With startpolicy='mandatory'
# virsh start vr-rhel6u3-x86_64-kvm
error: Failed to start domain vr-rhel6u3-x86_64-kvm
error: internal error Did not find USB device 951:1646

# virsh restore /tmp/vr-rhel6u3-x86_64-kvm.save
error: Failed to restore domain from /tmp/vr-rhel6u3-x86_64-kvm.save
error: internal error Did not find USB device 951:1646

# virsh migrate --live vr-rhel6u3-x86_64-kvm qemu+ssh://10.66.5.143/system --verbose
root.5.143's password: 
error: internal error Did not find USB device 951:1646

4. Check the --migratable in dumpxml
# virsh dumpxml vr-rhel6u3-x86_64-kvm --migratable
...
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source startupPolicy='mandatory' autoAddress='yes'>
        <vendor id='0x0951'/>
        <product id='0x1646'/>
        <address bus='2' device='41'/>
      </source>
      <alias name='hostdev0'/>
    </hostdev>
...
and the usb controller section will be removed


5. Migrate to target without host usb or restore guest without host usb, the xml in target changed to
 <hostdev mode='subsystem' type='usb' managed='yes'>
      <source startupPolicy='optional' missing='yes'>
        <vendor id='0x0951'/>
        <product id='0x1646'/>
        <address bus='2' device='41'/>
      </source>
      <alias name='hostdev0'/>
    </hostdev>

If have the usb device with the same verdor and product id on target, then the address change to new bus and device num and migration/restore can succeed with all 3 policies

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html