RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 805172 - Add live migration support for USB
Summary: Add live migration support for USB
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.4
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Gerd Hoffmann
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 809613 841629 875700 (view as bug list)
Depends On:
Blocks: 832177 798682 804161 838170 840699 840804 843560 852624
TreeView+ depends on / blocked
 
Reported: 2012-03-20 15:55 UTC by Andrew Cathrow
Modified: 2018-12-01 15:08 UTC (History)
19 users (show)

Fixed In Version: qemu-kvm-0.12.1.2-2.320.el6
Doc Type: Release Note
Doc Text:
Live Migration of Guests Using USB Devices In Red Hat Enterprise Linux 6.4, KVM supports live migration of guests with USB devices. The following devices are supported: Enhanced Host Controller Interface (EHCI) and Universal Host Controller Interface (UHCI) local passthrough and emulated devices such as storage devices, mice, keyboards, hubs, and others.
Clone Of:
: 838170 843560 (view as bug list)
Environment:
Last Closed: 2013-02-21 07:33:03 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:0527 0 normal SHIPPED_LIVE qemu-kvm bug fix and enhancement update 2013-02-20 21:51:08 UTC

Description Andrew Cathrow 2012-03-20 15:55:58 UTC 
The new USB implementation in qemu does not support live migration, even if no usb devices are being redirected.

We need to support live migration for usb redirection, with and without attached usb devices.
Comment 4 Gerd Hoffmann 2012-04-20 12:38:16 UTC 
It's ehci, usb-storage and usb-redir.
Maybe uhci needs some changes too.
Comment 5 Dave Allan 2012-06-06 14:35:45 UTC 
It doesn't look to me like there's any libvirt impact here.  Jiri, does it look that way to you?  If there is, can you clone the BZ?
Comment 7 Gerd Hoffmann 2012-07-03 10:34:10 UTC 
Patch series pulled.  uhci, ehci, usb-storage & usb-host can handle migration now in upstream qemu.  backport tdb asap.
Comment 9 Jiri Denemark 2012-07-25 14:57:38 UTC 
Gerd, how does this affect USB passthrough (i.e. host USB devices pass through to a guest)? Currently, libvirt refuses to migrate when a <hostdev> element is specified in domain XML, which of course includs <hostdev mode='subsystem' type='usb'>.
Comment 10 Gerd Hoffmann 2012-08-06 09:22:27 UTC 
(In reply to comment #9)
> Gerd, how does this affect USB passthrough (i.e. host USB devices pass
> through to a guest)? Currently, libvirt refuses to migrate when a <hostdev>
> element is specified in domain XML, which of course includs <hostdev
> mode='subsystem' type='usb'>.

There are no changes required of course.

Qemu handles live migration of usb passthrough devices just fine now, so it is possible for libvirt to allow live migration even with usb pass through devices being present.

Might be a non-trivial effort though.  As far I know libvirt can't handle usb host device hotplug and refuses to start the guest in case the host usb device isn't present.  Not fully sure what the reason is, probably libvirt wants update cgroups and selinux labels for /dev/usb/* files before starting qemu so it can actually access the usb device.

For live migration to work correctly the '-device usb-host, ...' line must be present on the target machine, mo matter whenever the device is actally there or not, otherwise the target qemu will bail out when it find the usb-host state info in the migration data stream.

live migration is probably not an important use case anyway, unless you have some robot to migrate the usb device from one host machine to the other too ;)

But for save/restore (which libvirt implements as live-migration-to-file) it 
would be quite nice if it works even with usb-host devices present.
Comment 12 Dave Allan 2012-08-30 17:27:04 UTC 
(In reply to comment #10)
> live migration is probably not an important use case anyway, unless you have
> some robot to migrate the usb device from one host machine to the other too
> ;)
> 
> But for save/restore (which libvirt implements as live-migration-to-file) it 
> would be quite nice if it works even with usb-host devices present.

I agree completely with the first point: live migrating a VM with a passed through USB device is kind of crazy.  On the second point about save and restore, I think the potential for grief is high enough from the possibility of the device being moved/removed on the host that this should be documented as a known limitation.
Comment 13 Dave Allan 2012-08-30 17:31:20 UTC 
Just to clarify: I believe this BZ is about usb redirection, not local USB passthrough, and for usb redirection, it's completely valid to migrate the VM.  Is that the fix that was committed?
Comment 14 Dave Allan 2012-08-30 17:32:48 UTC 
Jiri, if there's a redirected USB device, does libvirt prohibit migration?  If so, that restriction should be removed.
Comment 15 Gerd Hoffmann 2012-08-31 07:23:27 UTC 
The posted patch series covers ehci, usb-storage and usb-host (aka local pass-through).  usb-redir (usb redirection over tcp/spice) is still work in progress, but planned for 6.4 too.
Comment 19 Gerd Hoffmann 2012-09-24 06:35:28 UTC 
*** Bug 841629 has been marked as a duplicate of this bug. ***
Comment 20 Eduardo Habkost 2012-09-25 23:32:39 UTC 
New patches were submitted to this bug. Moving back to POST.
Comment 25 Qunfang Zhang 2012-10-15 08:46:47 UTC 
Reproduced on qemu-kvm-0.12.1.2-2.310.el6. Live migration failed for USB device.

1. uhci + usb host:
 -usb -device usb-host,hostbus=2,hostaddr=8,id=hostdev0

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:01.2/1/usb-host'

2. echi + usb host:
-device usb-ehci,id=ehci -device usb-host,hostbus=2,hostaddr=8,id=hostdev0

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

3. echi + usb emulated storage
 -device usb-ehci,id=ehci -device   usb-storage,drive=drive-usb-0-0,id=usb-0-0,removable=on,bus=ehci.0,port=1  -drive  file=/mnt/usb.qcow2,if=none,id=drive-usb-0-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=native

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

4. uchi + all kinds of emulated usb device (kbd,mouse,storage,hub)
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=input0 -device  usb-hub,bus=usb.0,port=2,id=hub -device usb-storage,bus=usb.0,port=2.4,drive=drive-usb-2-0,id=usb-2-0,removable=on  -drive file=/mnt/usb.qcow2,if=none,id=drive-usb-2-0,media=disk,format=qcow2,cache=none,werror=stop,rerror=stop,aio=threads -device usb-kbd,id=kbd,port=2.1 -device usb-mouse,id=mouse,port=2.2

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:01.2/2.4/usb-storage'

5. usb redirection:
 -device usb-ehci,id=ehci,addr=0x7 -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=3

(qemu) migrate -d tcp:10.66.6.67:5800
state blocked by non-migratable device '0000:00:07.0/ehci'

Verified on qemu-kvm-0.12.1.2-2.320.el6. 
Scenarios 1-4 passed, guest and its usb devices work well after migration.
But for scenario 5, still failed and blocked by bug 861331. So will verify this bug together with bug 861331.
Comment 26 Qunfang Zhang 2012-10-16 06:30:23 UTC 
Based on above comment 25, as bug 861331 is fixed on qemu-kvm-327. So I verified live migration with usbredir device. 

5. usb redirection test.

Steps:
(1) Boot guest with usb redirection devices.
 /usr/libexec/qemu-kvm -name rhel6.4 -M rhel6.4.0 -cpu Conroe -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -uuid 19fe9276-cf27-574b-5cfb-853a08165f9c -rtc base=utc -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/mnt/RHEL-Server-6.4-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:28:8d:97,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5930,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -monitor stdio  -readconfig /mnt/ich9-ehci-uhci.cfg -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0,debug=3  -chardev spicevmc,name=usbredir,id=usbredirchardev2 -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2,bus=ehci.0,debug=3   -chardev spicevmc,name=usbredir,id=usbredirchardev3  -device usb-redir,chardev=usbredirchardev3,id=usbredirdev3,bus=ehci.0,debug=3

(2) Boot the guest with listening mode on dst host.

(3) Open the guest with spicy tools provided by spice-gtk-tools.

(4) Select "Auto redirect newly plugged in USB devices" in "Option" menu. And then plug a USB disk to spice client host.

(5) Migrate guest.
(qemu) __com.redhat_spice_migrate_info $dst_host_ip $dst_host_port 
(qemu) migrate -d tcp:$dst_host_ip:5800

Results:
Guest migrated successfully and both the usb redirection devices work well after migration.

So this bug is verified pass.
Comment 28 Jiri Denemark 2012-10-18 14:42:01 UTC 
libvirt needed to be updated as well and the work is tracked by bug 843560.
Comment 30 Ademar Reis 2012-11-15 14:14:02 UTC 
Added a note about the live-migration of local/emulated devices (this bug) and also added a note to Bug 861331 about live migration of usb-forwarding devices (via spice). Text added there:

"In RHEL6.4, KVM supports live migration of guests using USB
forwarding via spice, while maintaining existing USB device
redirection for all configured devices."

Gerd, please correct me if there's anything missing.
Comment 31 Ademar Reis 2012-11-15 14:40:33 UTC 
*** Bug 875700 has been marked as a duplicate of this bug. ***
Comment 32 errata-xmlrpc 2013-02-21 07:33:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0527.html
Comment 33 Gerd Hoffmann 2013-04-15 09:16:04 UTC 
*** Bug 809613 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.