Red Hat Bugzilla – Bug 1138579
Migration failed with nec-usb-xhci from RHEL7. 0 to RHEL7.1
Last modified: 2015-03-05 04:54:39 EST
Description of problem: Migration failed with usb storage of nec-usb-xhci from RHEL7.0 to RHEL7.1. Version-Release number of selected component (if applicable): src host: qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 kernel-3.10.0-123.8.1.el7.x86_64 dst host: kernel-3.10.0-152.el7.x86_64 qemu-kvm-rhev-2.1.0-3.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. boot a rhel7.1 guest with nec-usb-xhci storage on src host # /usr/libexec/qemu-kvm -cpu SandyBridge \ -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 \ -M pc-i440fx-rhel7.0.0 \ -drive file=/mnt/rhel7-ga.qcow2_v3,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,id=data-disk2,bootindex=1 \ -device nec-usb-xhci,id=xhci,addr=0x12 \ -device usb-storage,drive=drive-usb-0-1,id=usb-0-1,removable=on,bus=xhci.0,port=1 \ -drive file=/mnt/usb-ehci,if=none,id=drive-usb-0-1,media=disk,format=qcow2 \ -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 \ -monitor stdio 2. boot the guest on dst host # /usr/libexec/qemu-kvm -cpu SandyBridge \ -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 \ -M pc-i440fx-rhel7.0.0 \ -drive file=/mnt/rhel7-ga.qcow2_v3,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,id=data-disk2,bootindex=1 \ -device nec-usb-xhci,id=xhci,addr=0x12 \ -device usb-storage,drive=drive-usb-0-1,id=usb-0-1,removable=on,bus=xhci.0,port=1 \ -drive file=/mnt/usb-ehci,if=none,id=drive-usb-0-1,media=disk,format=qcow2 \ -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 \ -incoming tcp:0:5800 3. do migration (qemu) migrate -d tcp:10.66.9.152:5800 Actual results: after step3, migration is failed and qemu-kvm quits on the dst host: (qemu) red_dispatcher_loadvm_commands: qemu: warning: error while loading state for instance 0x0 of device '0000:00:12.0/xhci' qemu-kvm: load of migration failed: Invalid argument Expected results: migration can be finished normally and qemu-kvm doesn't quit. Additional info:
I also test the following version, also hit this problem: src host: qemu-kvm-rhev-1.5.3-60.el7_0.7.x86_64 kernel-3.10.0-123.8.1.el7.x86_64 dst host: kernel-3.10.0-152.el7.x86_64 qemu-kvm-rhev-2.1.0-3.el7.x86_64
Reproduced; there's a possibility that it's already fixed by an upstream fix that's getting backported at the moment, although that xhci bug is listed as 'rare' where as this is fully repeatable.
Apparently not fixed by the bz 1093023/1136512 fix from Laszlo/Markus
vmstate_load_state: For xhci id=1 vmstate_load_state: field parent_obj vmstate_load_state: For PCIEDevice id=2 vmstate_load_state: field version_id vmstate_load_state: field config pci.c:get_pci_config_device EINVAL for mask/flags mismatch i=113 config[i]=a0 s->config[i]=0 s->cmask[i]=ff s->wmask[i]=0 s->w1cmask[i]=0 Bailing -22 for config in PCIEDevice Bailing -22 for parent_obj in xhci qemu: warning: error while loading state for instance 0x0 of device '0000:00:12.0/xhci' qemu-system-x86_64: load of migration failed: Invalid argument
lspci -v on rhel 7.0 is showing: lspci -v shows [90] MSI-X: Enable+ Count=16 Masked- [70] MSI: Enable- Count=1/16 Maskable- 64bit+ [a0] Express Endpoint, MSI 00 on rhev 7.1 I'm seeing: [90] MSI-X: Enable+ Count=16 Masked- [70] MSI: Enable- Count=1/16 Maskable- 64bit+ so it looks like the Express Endpoint config has been eaten.
posted fix upstream
Fix included in qemu-kvm-rhev-2.1.0-4.el7
Reproduce this bug using the following version: src host: qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 kernel-3.10.0-123.8.1.el7.x86_64 dst host: kernel-3.10.0-167.el7.x86_64 qemu-kvm-rhev-2.1.0-3.el7.x86_64 Steps to Reproduce: 1. boot a rhel7.1 guest with nec-usb-xhci storage on src host # /usr/libexec/qemu-kvm -cpu SandyBridge \ -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 \ -M pc-i440fx-rhel7.0.0 \ -drive file=/mnt/rhel7-ga.qcow2_v3,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,id=data-disk2,bootindex=1 \ -device nec-usb-xhci,id=xhci,addr=0x12 \ -device usb-storage,drive=drive-usb-0-1,id=usb-0-1,removable=on,bus=xhci.0,port=1 \ -drive file=/mnt/usb-ehci,if=none,id=drive-usb-0-1,media=disk,format=qcow2 \ -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 \ -monitor stdio 2. boot the guest on dst host # /usr/libexec/qemu-kvm -cpu SandyBridge \ -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 \ -M pc-i440fx-rhel7.0.0 \ -drive file=/mnt/rhel7-ga.qcow2_v3,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,id=data-disk2,bootindex=1 \ -device nec-usb-xhci,id=xhci,addr=0x12 \ -device usb-storage,drive=drive-usb-0-1,id=usb-0-1,removable=on,bus=xhci.0,port=1 \ -drive file=/mnt/usb-ehci,if=none,id=drive-usb-0-1,media=disk,format=qcow2 \ -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 \ -incoming tcp:0:5800 3. do migration (qemu) migrate -d tcp:10.66.9.152:5800 Actual results: after step3, migration is failed and qemu-kvm quits on the dst host: (qemu) red_dispatcher_loadvm_commands: qemu: warning: error while loading state for instance 0x0 of device '0000:00:12.0/xhci' qemu-kvm: load of migration failed: Invalid argument
Test this bug on an intel host using the following version: src host: qemu-kvm-rhev-1.5.3-60.el7ev.x86_64 kernel-3.10.0-123.8.1.el7.x86_64 dst host: kernel-3.10.0-167.el7.x86_64 qemu-kvm-rhev-2.1.0-4.el7.x86_64 Steps to Test: 1. boot a rhel7.1 guest with nec-usb-xhci storage on src host # /usr/libexec/qemu-kvm -cpu SandyBridge \ -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 \ -M pc-i440fx-rhel7.0.0 \ -drive file=/mnt/rhel7-ga.qcow2_v3,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,id=data-disk2,bootindex=1 \ -device nec-usb-xhci,id=xhci,addr=0x12 \ -device usb-storage,drive=drive-usb-0-1,id=usb-0-1,removable=on,bus=xhci.0,port=1 \ -drive file=/mnt/usb-ehci,if=none,id=drive-usb-0-1,media=disk,format=qcow2 \ -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 \ -monitor stdio 2. boot the guest on dst host # /usr/libexec/qemu-kvm -cpu SandyBridge \ -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 \ -M pc-i440fx-rhel7.0.0 \ -drive file=/mnt/rhel7-ga.qcow2_v3,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,id=data-disk2,bootindex=1 \ -device nec-usb-xhci,id=xhci,addr=0x12 \ -device usb-storage,drive=drive-usb-0-1,id=usb-0-1,removable=on,bus=xhci.0,port=1 \ -drive file=/mnt/usb-ehci,if=none,id=drive-usb-0-1,media=disk,format=qcow2 \ -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 \ -incoming tcp:0:5800 3. do migration (qemu) migrate -d tcp:10.66.9.152:5800 Actual results: Migration can be finished normally. Can create a file on the nec-usb-xhci storage using "dd if=/dev/zero of=file bs=1M count=1000".
Test this bug on an amd host using the following version: src host: qemu-kvm-rhev-1.5.3-60.el7_0.9.x86_64 kernel-3.10.0-163.el7.x86_64 dst host: kernel-3.10.0-167.el7.x86_64 qemu-kvm-rhev-2.1.0-4.el7.x86_64 Guest: Windows 8.1 32 bits Steps to Test: 1. boot a win8.1_32 guest with nec-usb-xhci storage on src host # /usr/libexec/qemu-kvm -cpu SandyBridge \ -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 \ -M pc-i440fx-rhel7.0.0 \ -drive file=/mnt/win8.1_32.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,id=data-disk2,bootindex=1 \ -device nec-usb-xhci,id=xhci,addr=0x12 \ -device usb-storage,drive=drive-usb-0-1,id=usb-0-1,removable=on,bus=xhci.0,port=1 \ -drive file=/mnt/usb-ehci,if=none,id=drive-usb-0-1,media=disk,format=qcow2 \ -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 \ -monitor stdio 2. boot the guest on dst host # /usr/libexec/qemu-kvm -cpu SandyBridge \ -enable-kvm -m 4096 -realtime mlock=off -smp 4,sockets=2,cores=2,threads=1,maxcpus=160 -numa node,cpus=0 \ -M pc-i440fx-rhel7.0.0 \ -drive file=/mnt/win8.1_32.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=writethrough,werror=stop,rerror=stop \ -device virtio-scsi-pci,id=scsi0,addr=0x13 \ -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,id=data-disk2,bootindex=1 \ -device nec-usb-xhci,id=xhci,addr=0x12 \ -device usb-storage,drive=drive-usb-0-1,id=usb-0-1,removable=on,bus=xhci.0,port=1 \ -drive file=/mnt/usb-ehci,if=none,id=drive-usb-0-1,media=disk,format=qcow2 \ -spice port=5901,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.vram_size=33554432 \ -incoming tcp:0:5800 3. do migration (qemu) migrate -d tcp:10.66.106.21:5800 Actual results: Migration can be finished normally. Can create a file on the nec-usb-xhci storage.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0624.html