Hide Forgot
Description of problem: Migrate failed with virtio-vgpu device from RHEL-AV 8.0.0 to RHEL-AV 8.3.0 Version-Release number of selected component (if applicable): source host: kernel-4.18.0-80.29.1.el8_0.x86_64 qemu-kvm-3.1.0-20.module+el8.0.0.z+3438+2851622e.1.x86_64 destination host: kernel-4.18.0-240.1.1.el8_3.x86_64 qemu-kvm-5.1.0-14.module+el8.3.0+8438+644aff69.x86_64 How reproducible: 3/3 Steps to Reproduce: 1.Boot qemu command in source host /usr/libexec/qemu-kvm -M pc-q35-rhel8.0.0 -device virtio-vga -monitor stdio 2.Boot qemu command in destination host /usr/libexec/qemu-kvm -M pc-q35-rhel8.0.0 -device virtio-vga -monitor stdio -incoming defer 3.Migrate from source host to destination host Actual results: Migrate failed with following error (qemu) qemu-kvm: get_pci_config_device: Bad config data: i=0x34 read: 84 device: 98 cmask: ff wmask: 0 w1cmask:0 qemu-kvm: Failed to load PCIDevice:config qemu-kvm: Failed to load virtio-gpu:virtio qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0/virtio-gpu' qemu-kvm: warning: TSC frequency mismatch between VM (2397221 kHz) and host (1995191 kHz), and TSC scaling unavailable qemu-kvm: load of migration failed: Invalid argument Expected results: Migrate successfully Additional info:
> (qemu) qemu-kvm: get_pci_config_device: Bad config data: i=0x34 read: 84 #define PCI_CAPABILITY_LIST 0x34 /* Offset of first capability list entry */ Hmm, looks like something in pci (adding mst to cc) ... What does "lspci -vxxx" print for the virtio-gpu device (as root, both qemu versions please)?
'info pci' from the two cases: 3.1.0: Bus 0, device 2, function 0: VGA controller: PCI device 1af4:1050 PCI subsystem 1af4:1100 IRQ 11. BAR0: 32 bit prefetchable memory at 0xfe000000 [0xfe7fffff]. BAR2: 64 bit prefetchable memory at 0xfe800000 [0xfe803fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" 5.1.0 Bus 0, device 2, function 0: VGA controller: PCI device 1af4:1050 PCI subsystem 1af4:1100 IRQ 11, pin A BAR0: 32 bit prefetchable memory at 0xfe000000 [0xfe7fffff]. BAR2: 64 bit prefetchable memory at 0xfe800000 [0xfe803fff]. BAR4: 32 bit memory at 0xfebd4000 [0xfebd4fff]. BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe]. id "" we seem to have gained a BAR.
It looks like the problem here is MSI-X; the full error is: (qemu) qemu-kvm: get_pci_config_device: Bad config data: i=0x34 read: 98 device: 84 cmask: ff wmask: 0 w1cmask:0 lspci -vxxx from 5.1.0: 00:02.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01) (prog-if 00 [VGA controller]) Subsystem: Red Hat, Inc. Device 1100 Flags: bus master, fast devsel, latency 0, IRQ 22 Memory at fe000000 (32-bit, prefetchable) [size=8M] Memory at fe800000 (64-bit, prefetchable) [size=16K] Memory at febd4000 (32-bit, non-prefetchable) [size=4K] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [98] MSI-X: Enable+ Count=3 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: <unknown> Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci 00: f4 1a 50 10 07 05 10 00 01 00 00 03 00 00 00 00 10: 08 00 00 fe 00 00 00 00 0c 00 80 fe 00 00 00 00 20: 00 40 bd fe 00 00 00 00 00 00 00 00 f4 1a 00 11 30: 00 00 bc fe 98 00 00 00 00 00 00 00 0b 01 00 00 40: 09 00 10 01 02 00 00 00 00 10 00 00 00 08 00 00 50: 09 40 10 03 02 00 00 00 00 18 00 00 00 08 00 00 60: 09 50 10 04 02 00 00 00 00 20 00 00 00 10 00 00 70: 09 60 14 02 02 00 00 00 00 30 00 00 00 10 00 00 80: 04 00 00 00 09 70 14 05 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 11 84 02 80 04 00 00 00 a0: 04 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 lspci from 3.1.0 00:02.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01) (prog-if 00 [VGA controller]) Subsystem: Red Hat, Inc. Device 1100 Flags: bus master, fast devsel, latency 0, IRQ 22 Memory at fe000000 (32-bit, prefetchable) [size=8M] Memory at fe800000 (64-bit, prefetchable) [size=16K] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [84] Vendor Specific Information: VirtIO: <unknown> Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci 00: f4 1a 50 10 07 01 10 00 01 00 00 03 00 00 00 00 10: 08 00 00 fe 00 00 00 00 0c 00 80 fe 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 f4 1a 00 11 30: 00 00 bc fe 84 00 00 00 00 00 00 00 0b 01 00 00 40: 09 00 10 01 02 00 00 00 00 10 00 00 00 08 00 00 50: 09 40 10 03 02 00 00 00 00 18 00 00 00 08 00 00 60: 09 50 10 04 02 00 00 00 00 20 00 00 00 10 00 00 70: 09 60 14 02 02 00 00 00 00 30 00 00 00 10 00 00 80: 04 00 00 00 09 70 14 05 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 the 98/84 is the start of the chain, and we've gained an MSI-X capability at 98.
This looks to me as if it happened somewhere between 4.0.0 and 4.1.0
(In reply to Dr. David Alan Gilbert from comment #5) > It looks like the problem here is MSI-X: > the 98/84 is the start of the chain, and we've gained an MSI-X capability at > 98. Yep. Seems the default for vectors= changed from 0 to 3.
Bisect landed at: commit c68082c43a3ddeb5e5da4ab401e3f9f422e7a290 Author: Marc-André Lureau <marcandre.lureau@redhat.com> Date: Fri May 24 15:09:45 2019 +0200 virtio-gpu: split virtio-gpu-pci & virtio-vga Add base classes that are common to vhost-user-gpu-pci and vhost-user-vga. Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Message-id: 20190524130946.31736-9-marcandre.lureau@redhat.com Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> Also it's not that the vectors default changed. The vectors property simply isn't there in v4.0. It isn't obvious why, it should have been there. Beside that I don't see an easy way to fix that given that both 3.1 (vectors=0) and 4.1 (vectors=3) are released already. We can make 5.1 compatible with the one or the other but not both.
Yes; so I think we need to close this as 'cantfix' because as you say we can fix it one way or the other; and given the bad choices we may as well keep compatibility with the newer version. I'm guessing this affects a range of !av versions as well.