Bug 805172

It's ehci, usb-storage and usb-redir.
Maybe uhci needs some changes too.

It doesn't look to me like there's any libvirt impact here.  Jiri, does it look that way to you?  If there is, can you clone the BZ?

Patch series pulled.  uhci, ehci, usb-storage & usb-host can handle migration now in upstream qemu.  backport tdb asap.

Gerd, how does this affect USB passthrough (i.e. host USB devices pass through to a guest)? Currently, libvirt refuses to migrate when a <hostdev> element is specified in domain XML, which of course includs <hostdev mode='subsystem' type='usb'>.

(In reply to comment #9)
> Gerd, how does this affect USB passthrough (i.e. host USB devices pass
> through to a guest)? Currently, libvirt refuses to migrate when a <hostdev>
> element is specified in domain XML, which of course includs <hostdev
> mode='subsystem' type='usb'>.

There are no changes required of course.

Qemu handles live migration of usb passthrough devices just fine now, so it is possible for libvirt to allow live migration even with usb pass through devices being present.

Might be a non-trivial effort though.  As far I know libvirt can't handle usb host device hotplug and refuses to start the guest in case the host usb device isn't present.  Not fully sure what the reason is, probably libvirt wants update cgroups and selinux labels for /dev/usb/* files before starting qemu so it can actually access the usb device.

For live migration to work correctly the '-device usb-host, ...' line must be present on the target machine, mo matter whenever the device is actally there or not, otherwise the target qemu will bail out when it find the usb-host state info in the migration data stream.

live migration is probably not an important use case anyway, unless you have some robot to migrate the usb device from one host machine to the other too ;)

But for save/restore (which libvirt implements as live-migration-to-file) it 
would be quite nice if it works even with usb-host devices present.

(In reply to comment #10)
> live migration is probably not an important use case anyway, unless you have
> some robot to migrate the usb device from one host machine to the other too
> ;)
> 
> But for save/restore (which libvirt implements as live-migration-to-file) it 
> would be quite nice if it works even with usb-host devices present.

I agree completely with the first point: live migrating a VM with a passed through USB device is kind of crazy.  On the second point about save and restore, I think the potential for grief is high enough from the possibility of the device being moved/removed on the host that this should be documented as a known limitation.

Just to clarify: I believe this BZ is about usb redirection, not local USB passthrough, and for usb redirection, it's completely valid to migrate the VM.  Is that the fix that was committed?

Jiri, if there's a redirected USB device, does libvirt prohibit migration?  If so, that restriction should be removed.

The posted patch series covers ehci, usb-storage and usb-host (aka local pass-through).  usb-redir (usb redirection over tcp/spice) is still work in progress, but planned for 6.4 too.

*** Bug 841629 has been marked as a duplicate of this bug. ***

New patches were submitted to this bug. Moving back to POST.

Reproduced on qemu-kvm-0.12.1.2-2.310.el6. Live migration failed for USB device.

1. uhci + usb host:
 -usb -device usb-host,hostbus=2,hostaddr=8,id=hostdev0

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:01.2/1/usb-host'

2. echi + usb host:
-device usb-ehci,id=ehci -device usb-host,hostbus=2,hostaddr=8,id=hostdev0

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

3. echi + usb emulated storage
 -device usb-ehci,id=ehci -device   usb-storage,drive=drive-usb-0-0,id=usb-0-0,removable=on,bus=ehci.0,port=1  -drive  file=/mnt/usb.qcow2,if=none,id=drive-usb-0-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

4. uchi + all kinds of emulated usb device (kbd,mouse,storage,hub)
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=input0 -device  usb-hub,bus=usb.0,port=2,id=hub -device usb-storage,bus=usb.0,port=2.4,drive=drive-usb-2-0,id=usb-2-0,removable=on  -drive file=/mnt/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads -device usb-kbd,id=kbd,port=2.1 -device usb-mouse,id=mouse,port=2.2

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:01.2/2.4/usb-storage'

5. usb redirection:
 -device usb-ehci,id=ehci,addr=0x7 -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=3

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

Verified on qemu-kvm-0.12.1.2-2.320.el6. 
Scenarios 1-4 passed, guest and its usb devices work well after migration.
But for scenario 5, still failed and blocked by bug 861331. So will verify this bug together with bug 861331.

Based on above comment 25, as bug 861331 is fixed on qemu-kvm-327. So I verified live migration with usbredir device. 

5. usb redirection test.

Steps:
(1) Boot guest with usb redirection devices.
 /usr/libexec/qemu-kvm -name rhel6.4 -M rhel6.4.0 -cpu Conroe -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -uuid 19fe9276-cf27-574b-5cfb-853a08165f9c -rtc base=utc -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/mnt/RHEL-Server-6.4-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:28:8d:97,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5930,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -monitor stdio  -readconfig /mnt/ich9-ehci-uhci.cfg -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=3  -chardev spicevmc,name=usbredir,id=usbredirchardev2 -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2,bus=ehci.0,debug=3   -chardev spicevmc,name=usbredir,id=usbredirchardev3  -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3,bus=ehci.0,debug=3

(2) Boot the guest with listening mode on dst host.

(3) Open the guest with spicy tools provided by spice-gtk-tools.

(4) Select "Auto redirect newly plugged in USB devices" in "Option" menu. And then plug a USB disk to spice client host.

(5) Migrate guest.
(qemu) __com.redhat_spice_migrate_info $dst_host_ip $dst_host_port 
(qemu) migrate -d tcp:$dst_host_ip:5800

Results:
Guest migrated successfully and both the usb redirection devices work well after migration.

So this bug is verified pass.

libvirt needed to be updated as well and the work is tracked by bug 843560.

Added a note about the live-migration of local/emulated devices (this bug) and also added a note to Bug 861331 about live migration of usb-forwarding devices (via spice). Text added there:

"In RHEL6.4, KVM supports live migration of guests using USB
forwarding via spice, while maintaining existing USB device
redirection for all configured devices."

Gerd, please correct me if there's anything missing.

*** Bug 875700 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0527.html

*** Bug 809613 has been marked as a duplicate of this bug. ***