Bug 805172 - Add live migration support for USB
Add live migration support for USB
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.4
All Linux
high Severity high
: rc
: ---
Assigned To: Gerd Hoffmann
Virtualization Bugs
:
: 809613 841629 875700 (view as bug list)
Depends On:
Blocks: 832177 798682 804161 838170 840699 840804 843560 852624
  Show dependency treegraph
 
Reported: 2012-03-20 11:55 EDT by Andrew Cathrow
Modified: 2014-09-07 18:54 EDT (History)
19 users (show)

See Also:
Fixed In Version: qemu-kvm-0.12.1.2-2.320.el6
Doc Type: Release Note
Doc Text:
Live Migration of Guests Using USB Devices In Red Hat Enterprise Linux 6.4, KVM supports live migration of guests with USB devices. The following devices are supported: Enhanced Host Controller Interface (EHCI) and Universal Host Controller Interface (UHCI) local passthrough and emulated devices such as storage devices, mice, keyboards, hubs, and others.
Story Points: ---
Clone Of:
: 838170 843560 (view as bug list)
Environment:
Last Closed: 2013-02-21 02:33:03 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:0527 normal SHIPPED_LIVE qemu-kvm bug fix and enhancement update 2013-02-20 16:51:08 EST

  None (edit)
Description Andrew Cathrow 2012-03-20 11:55:58 EDT
The new USB implementation in qemu does not support live migration, even if no usb devices are being redirected.

We need to support live migration for usb redirection, with and without attached usb devices.
Comment 4 Gerd Hoffmann 2012-04-20 08:38:16 EDT
It's ehci, usb-storage and usb-redir.
Maybe uhci needs some changes too.
Comment 5 Dave Allan 2012-06-06 10:35:45 EDT
It doesn't look to me like there's any libvirt impact here.  Jiri, does it look that way to you?  If there is, can you clone the BZ?
Comment 7 Gerd Hoffmann 2012-07-03 06:34:10 EDT
Patch series pulled.  uhci, ehci, usb-storage & usb-host can handle migration now in upstream qemu.  backport tdb asap.
Comment 9 Jiri Denemark 2012-07-25 10:57:38 EDT
Gerd, how does this affect USB passthrough (i.e. host USB devices pass through to a guest)? Currently, libvirt refuses to migrate when a <hostdev> element is specified in domain XML, which of course includs <hostdev mode='subsystem' type='usb'>.
Comment 10 Gerd Hoffmann 2012-08-06 05:22:27 EDT
(In reply to comment #9)
> Gerd, how does this affect USB passthrough (i.e. host USB devices pass
> through to a guest)? Currently, libvirt refuses to migrate when a <hostdev>
> element is specified in domain XML, which of course includs <hostdev
> mode='subsystem' type='usb'>.

There are no changes required of course.

Qemu handles live migration of usb passthrough devices just fine now, so it is possible for libvirt to allow live migration even with usb pass through devices being present.

Might be a non-trivial effort though.  As far I know libvirt can't handle usb host device hotplug and refuses to start the guest in case the host usb device isn't present.  Not fully sure what the reason is, probably libvirt wants update cgroups and selinux labels for /dev/usb/* files before starting qemu so it can actually access the usb device.

For live migration to work correctly the '-device usb-host, ...' line must be present on the target machine, mo matter whenever the device is actally there or not, otherwise the target qemu will bail out when it find the usb-host state info in the migration data stream.

live migration is probably not an important use case anyway, unless you have some robot to migrate the usb device from one host machine to the other too ;)

But for save/restore (which libvirt implements as live-migration-to-file) it 
would be quite nice if it works even with usb-host devices present.
Comment 12 Dave Allan 2012-08-30 13:27:04 EDT
(In reply to comment #10)
> live migration is probably not an important use case anyway, unless you have
> some robot to migrate the usb device from one host machine to the other too
> ;)
> 
> But for save/restore (which libvirt implements as live-migration-to-file) it 
> would be quite nice if it works even with usb-host devices present.

I agree completely with the first point: live migrating a VM with a passed through USB device is kind of crazy.  On the second point about save and restore, I think the potential for grief is high enough from the possibility of the device being moved/removed on the host that this should be documented as a known limitation.
Comment 13 Dave Allan 2012-08-30 13:31:20 EDT
Just to clarify: I believe this BZ is about usb redirection, not local USB passthrough, and for usb redirection, it's completely valid to migrate the VM.  Is that the fix that was committed?
Comment 14 Dave Allan 2012-08-30 13:32:48 EDT
Jiri, if there's a redirected USB device, does libvirt prohibit migration?  If so, that restriction should be removed.
Comment 15 Gerd Hoffmann 2012-08-31 03:23:27 EDT
The posted patch series covers ehci, usb-storage and usb-host (aka local pass-through).  usb-redir (usb redirection over tcp/spice) is still work in progress, but planned for 6.4 too.
Comment 19 Gerd Hoffmann 2012-09-24 02:35:28 EDT
*** Bug 841629 has been marked as a duplicate of this bug. ***
Comment 20 Eduardo Habkost 2012-09-25 19:32:39 EDT
New patches were submitted to this bug. Moving back to POST.
Comment 25 Qunfang Zhang 2012-10-15 04:46:47 EDT
Reproduced on qemu-kvm-0.12.1.2-2.310.el6. Live migration failed for USB device.

1. uhci + usb host:
 -usb -device usb-host,hostbus=2,hostaddr=8,id=hostdev0

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:01.2/1/usb-host'

2. echi + usb host:
-device usb-ehci,id=ehci -device usb-host,hostbus=2,hostaddr=8,id=hostdev0

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

3. echi + usb emulated storage
 -device usb-ehci,id=ehci -device   usb-storage,drive=drive-usb-0-0,id=usb-0-0,removable=on,bus=ehci.0,port=1  -drive  file=/mnt/usb.qcow2,if=none,id=drive-usb-0-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

4. uchi + all kinds of emulated usb device (kbd,mouse,storage,hub)
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=input0 -device  usb-hub,bus=usb.0,port=2,id=hub -device usb-storage,bus=usb.0,port=2.4,drive=drive-usb-2-0,id=usb-2-0,removable=on  -drive file=/mnt/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads -device usb-kbd,id=kbd,port=2.1 -device usb-mouse,id=mouse,port=2.2

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:01.2/2.4/usb-storage'

5. usb redirection:
 -device usb-ehci,id=ehci,addr=0x7 -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=3

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

Verified on qemu-kvm-0.12.1.2-2.320.el6. 
Scenarios 1-4 passed, guest and its usb devices work well after migration.
But for scenario 5, still failed and blocked by bug 861331. So will verify this bug together with bug 861331.
Comment 26 Qunfang Zhang 2012-10-16 02:30:23 EDT
Based on above comment 25, as bug 861331 is fixed on qemu-kvm-327. So I verified live migration with usbredir device. 

5. usb redirection test.

Steps:
(1) Boot guest with usb redirection devices.
 /usr/libexec/qemu-kvm -name rhel6.4 -M rhel6.4.0 -cpu Conroe -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -uuid 19fe9276-cf27-574b-5cfb-853a08165f9c -rtc base=utc -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/mnt/RHEL-Server-6.4-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:28:8d:97,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5930,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -monitor stdio  -readconfig /mnt/ich9-ehci-uhci.cfg -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=3  -chardev spicevmc,name=usbredir,id=usbredirchardev2 -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2,bus=ehci.0,debug=3   -chardev spicevmc,name=usbredir,id=usbredirchardev3  -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3,bus=ehci.0,debug=3

(2) Boot the guest with listening mode on dst host.

(3) Open the guest with spicy tools provided by spice-gtk-tools.

(4) Select "Auto redirect newly plugged in USB devices" in "Option" menu. And then plug a USB disk to spice client host.

(5) Migrate guest.
(qemu) __com.redhat_spice_migrate_info $dst_host_ip $dst_host_port 
(qemu) migrate -d tcp:$dst_host_ip:5800

Results:
Guest migrated successfully and both the usb redirection devices work well after migration.

So this bug is verified pass.
Comment 28 Jiri Denemark 2012-10-18 10:42:01 EDT
libvirt needed to be updated as well and the work is tracked by bug 843560.
Comment 30 Ademar Reis 2012-11-15 09:14:02 EST
Added a note about the live-migration of local/emulated devices (this bug) and also added a note to Bug 861331 about live migration of usb-forwarding devices (via spice). Text added there:

"In RHEL6.4, KVM supports live migration of guests using USB
forwarding via spice, while maintaining existing USB device
redirection for all configured devices."

Gerd, please correct me if there's anything missing.
Comment 31 Ademar Reis 2012-11-15 09:40:33 EST
*** Bug 875700 has been marked as a duplicate of this bug. ***
Comment 32 errata-xmlrpc 2013-02-21 02:33:03 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0527.html
Comment 33 Gerd Hoffmann 2013-04-15 05:16:04 EDT
*** Bug 809613 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.