Bug 1353180

Summary: 7.3->7.2 migration: qemu-kvm: usbredirparser: error unserialize caps mismatch
Product: Red Hat Enterprise Linux 7 Reporter: Dr. David Alan Gilbert <dgilbert>
Component: qemu-kvm-rhevAssignee: Gerd Hoffmann <kraxel>
Status: CLOSED ERRATA QA Contact: huiqingding <huding>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: chayang, dgilbert, hdegoede, huding, juzhang, knoel, kraxel, mrezanin, qizhu, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.6.0-16.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-07 21:21:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1289197    

Description Dr. David Alan Gilbert 2016-07-06 12:01:43 UTC
Description of problem:

migrating from a 7.3 host to a 7.2 host gives the error in the destination qemu log:

qemu-kvm: usb-redir warning: usb-redir connection broken during migration
qemu-kvm: usbredirparser: error unserialize caps mismatch
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:06.7/2/usb-redir'
qemu-kvm: load of migration failed: Operation not permitted

I suspect the problem here is that in qemu's usbredir_create_parser we have:
#if USBREDIR_VERSION >= 0x000700
    usbredirparser_caps_set_cap(caps, usb_redir_cap_bulk_streams);
#endif

now since 7.3 seems to have the newer USBREDIR we've suddenly triggered that code.  I suspect it needs tieing to machine type (i.e. only 7.3 or newer) but there's a lot of places that ifdef happens so I don't know what the consequences are.

Version-Release number of selected component (if applicable):
source:
qemu-kvm-rhev-2.6.0-11.el7.1353070a.x86_64
usbredir-0.7.1-1.el7.x86_64

destination:
qemu-kvm-rhev-2.3.0-31.el7_2.16.x86_64
usbredir-0.6-7.el7.x86_64

How reproducible:
100%?

Steps to Reproduce:
1. Migrate from a 7.3 host to a 7.2 host
destination command line in this case is the following; but I think you just need a usbredir device.

/usr/libexec/qemu-kvm -name guest=7.2-test,debug-threads=on -S -machine rhel6.6.0,accel=kvm,usb=off,vmport=off -cpu IvyBridge -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid b243112a-8767-4304-ac82-4b40a90449a3 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-7.2-test/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0xb -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/home/vms/7.2a.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:ae:2d:0e,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -device AC97,id=sound0,bus=pci.0,addr=0x4 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on


Actual results:


Expected results:


Additional info:

Comment 1 Dr. David Alan Gilbert 2016-07-07 14:13:46 UTC
This is related to bz 1185167 - but does this fix us for it? It still seems right to tie the new feature to the new machine type.

Comment 2 Gerd Hoffmann 2016-07-13 11:13:50 UTC
> I suspect the problem here is that in qemu's usbredir_create_parser we have:
> #if USBREDIR_VERSION >= 0x000700
>     usbredirparser_caps_set_cap(caps, usb_redir_cap_bulk_streams);
> #endif

> now since 7.3 seems to have the newer USBREDIR we've suddenly triggered that
> code.  I suspect it needs tieing to machine type (i.e. only 7.3 or newer)

Yes.

> but there's a lot of places that ifdef happens so I don't know what the
> consequences are.

Should be easy, the caps thing is there exactly so you can turn it off.  And all the streams support code has both compile time (#ifdef) and runtime (cap) checks for the same reason.  The only thing missing is a property so you can actually turn it off.

http://git.engineering.redhat.com/git/users/ghoffman/rhel7/qemu-kvm-rhev.git/log/?h=bz1353180-usbredir-migration

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=11343436

Can you give it a try?

Comment 3 Gerd Hoffmann 2016-07-13 11:31:55 UTC
https://patchwork.ozlabs.org/patch/647864/

Comment 4 Dr. David Alan Gilbert 2016-07-13 15:41:00 UTC
OK, that passes my smoke test of migrating; wasn't trying to do anything usb related or streams related though.

Dave

Comment 5 Miroslav Rezanina 2016-07-26 06:57:03 UTC
Fix included in qemu-kvm-rhev-2.6.0-16.el7

Comment 7 huiqingding 2016-09-13 09:47:30 UTC
Reproduce this bug using:
7.3 host:
kernel-3.10.0-505.el7.x86_64
qemu-kvm-rhev-2.6.0-12.el7.x86_64
usbredir-0.7.1-1.el7.x86_64
7.2 host:
kernel-3.10.0-327.37.1.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.21.x86_64
usbredir-0.6-7.el7.x86_64

Reproduce steps:
1. boot a rhel7.3 guest on source 7.3 and destination 7.2 host
# /usr/libexec/qemu-kvm -name guest=7.2-test,debug-threads=on -S -machine rhel6.6.0,accel=kvm,usb=off,vmport=off -cpu IvyBridge -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid b243112a-8767-4304-ac82-4b40a90449a3 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-7.2-test/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0xb -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/mnt/rhel7.3.qcow2,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:ae:2d:0e,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -device AC97,id=sound0,bus=pci.0,addr=0x4 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on -monitor stdio

2. do migration from 7.3 to 7.2

Results:
after step2, qemu-kvm of 7.2 quits with error:
2016-09-13T09:28:50.270999Z qemu-kvm: usbredirparser: error unserialize caps mismatch
2016-09-13T09:28:50.271040Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:06.7/2/usb-redir'
2016-09-13T09:28:50.271097Z qemu-kvm: load of migration failed: Operation not permitted

Verify this bug using:
7.3 host:
kernel-3.10.0-505.el7.x86_64
qemu-kvm-rhev-2.6.0-24.el7.x86_64
usbredir-0.7.1-1.el7.x86_64
7.2 host:
kernel-3.10.0-327.37.1.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.21.x86_64
usbredir-0.6-7.el7.x86_64

after step2, migration can be finished normally.

Comment 8 huiqingding 2016-09-13 09:48:31 UTC
Based on comment #7, set this bug to be verified.

Comment 10 errata-xmlrpc 2016-11-07 21:21:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html