According to the previous test logs, it should be a regression bug, and it's not reproducible on the previous builds
On RHEL 9.0.0 host,
qemu-kvm-6.2.0-11.el9_0.5
On RHEL9.2.0 host,
qemu-kvm-7.1.0-5.el9.x86_64

This bug blocks the stable guest abi from rhel8.x to rhel 9.3.

I have an update on the investigation an a fix:

(In reply to Min Deng from comment #0)
> [...]
> 1.boot up a guest with the following cmd
> /usr/libexec/qemu-kvm -name mouse-vm -sandbox on -machine [...]
> stdio -watchdog-action reset

I could reduce this cmdline to an smaller & simpler version. The focus points here are:

-machine pc-q35-rhel8.5.0
-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \
-device driver=nec-usb-xhci,p2=4,p3=4,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1 

I could reproduce this same issue on upstream qemu (both host and guest) switching the machine type:
-machine pc-q35-6.0 \

The same did not reproduce in machine type >= pc-q35-6.1. So I did some hacking to find out the different feature is this:
{ "ICH9-LPC", ACPI_PM_PROP_ACPI_PCIHP_BRIDGE, "off" }

If we change this to "on", the bug does not reproduce.

What does it change:
When the PCIDevice nec-usb-xhci gets added in the VM, the bus above (pcie-root-port18) starts a hotplug action:
- "on": hotplug_handler_plug calls ich9_pm_device_plug_cb(), 
- "off": hotplug_handler_plug calls pcie_cap_slot_plug_cb(), which sets config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI hotplug for the guest.

After a while, in "off" case, the guest will deal with this hotplug and qemu will clear above bit.

About the bug:
> (qemu) qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0
> device: 40 cmask: ff wmask: 0 w1cmask:19
> qemu-kvm: Failed to load PCIDevice:config
> qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
> qemu-kvm: error while loading state for instance 0x0 of device
> '0000:00:12.0/pcie-root-port'

As above message states, it happens in get_pci_config_device(): 
this routine seems to verify if the device received via migration is in a config state compatible with the one created in the target qemu.

It fails because on the received device, config byte 0x6e has value 0x00, while on the target qemu (freshly created) the value is 0x40 (PCI_EXP_SLTSTA_PDS set).
This happens because while the source guest was able to clear this bit, the freshly created qemu target have not.

So it makes sense to skip checking this bit, and avoid this issue.
For this, IIUC it makes sense to use the device's PCIDevice->w1cmask, which is used to implement RW1C(Write 1 to Clear) bytes, and flags which bits of 
PCIDevice->config should be skipped in get_pci_config_device().

On pcie_cap_slot_init() the PCI_EXP_SLTSTA (slot status) is already set with some bits, so make use of it to also set PCI_EXP_SLTSTA_PDS on it.

For me, it seems to completely fix the bug, so I backported to rhel9.3 and created a brew for further testing:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=53635324

I am currently working on the patch commit message, and will send it soon.

Thank for leobras's effort, the test for above build has started and will update this bug once I get the result.

Upstream patch v1:
https://lore.kernel.org/qemu-devel/20230629090500.438976-2-leobras@redhat.com/

QE tried to test stable guest abi from rhel9.2 to rhel9.3 with leonardo's patch installed, it seems it fixed the forward migration but there's still the issue in the backward migration
src:
rhel9.2
kernel-5.14.0-284.18.1.el9_2.x86_64
qemu-kvm-7.2.0-14.el9_2.1.x86_64
edk2-ovmf-20221207gitfff6d81270b5-9.el9_2.noarch

dst:
RHEL 9.3
kernel-5.14.0-325.el9.x86_64
qemu-kvm-8.0.0-6.el9.leonardo202306290450.x86_64
edk2-ovmf-20230301gitf80f052277c8-5.el9.noarch

Steps
QE usually do ping-ping migration in our test cases.

1.migrate guest from src to dst  - successfully (but need to wait a while and then the vm was activated on dst side)

2.migrate the guest back from dst to src

Actually result:
It occurs on rhel9.2(src) host
(qemu) migrate_incoming tcp:[::]:4000
(qemu) qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument

Expected result: 
The guest works well

Any issues please let me know ! Thanks


cmdline,
/usr/libexec/qemu-kvm -name "mouse-vm" -sandbox on -machine pc-q35-rhel8.3.0,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars -nodefaults  -cpu Cascadelake-Server-noTSX,+kvm_pv_unhalt,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt -chardev socket,id=charmonitor,path=/home/tmp1,server=on,wait=off -chardev pty,id=charserial0 -chardev socket,id=charchannel1,path=/home/tmp2,server=on,wait=off -chardev socket,id=seabioslog_id,path=/home/seabios,server=on,wait=off -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device ahci,id=ahci0,bus=pcie.0,addr=0x3 -device ide-cd,bus=ahci0.0,unit=0,drive=drive-ide0-1-1,id=ide0-1-1 -device '{"driver":"pcie-root-port","port":1,"chassis":1,"id":"pcie-root-port0","multifunction":true,"bus":"pcie.0","addr":"0x4"}' -device '{"driver":"virtio-scsi-pci","id":"scsi0","bus":"pcie-root-port0"}' -device '{"driver":"pcie-root-port","port":2,"chassis":2,"id":"pcie-root-port1","bus":"pcie.0","addr":"0x4.0x1"}' -device '{"driver":"scsi-hd","bus":"scsi0.0","lun":0,"drive":"drive-virtio-disk0","id":"virtio-disk0","bootindex":1}' -device '{"driver":"scsi-hd","drive":"drive-scsi-disk","bus":"scsi0.0","lun":1,"id":"data-disk1"}' -device '{"driver":"pcie-root-port","port":3,"chassis":3,"id":"pcie-root-port2","bus":"pcie.0","addr":"0x4.0x2"}' -device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pcie-root-port2"}' -device '{"driver":"isa-serial","id":"serial0","chardev":"charserial0"}' -device intel-hda,id=sound0,bus=pcie.0,addr=0x7 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device intel-hda,id=sound1,bus=pcie.0,addr=0x8 -device hda-micro,id=sound1-codec0,bus=sound1.0 -device intel-hda,id=sound2,bus=pcie.0,addr=0x9 -device hda-output,id=sound2-codec0,bus=sound2.0,cad=0 -device ich9-intel-hda,id=sound3,bus=pcie.0,addr=0xa -device hda-duplex,id=sound3-codec0,bus=sound3.0,cad=0 -device '{"driver":"pvpanic","ioport":1285}' -device '{"driver":"pcie-root-port","port":4,"chassis":4,"id":"pcie-root-port3","bus":"pcie.0","addr":"0x4.0x3"}' -device '{"driver":"e1000e","netdev":"hostnet1","id":"virtio-net-pci1","mac":"00:52:68:26:31:03","bus":"pcie-root-port3"}' -device '{"driver":"pcie-root-port","port":5,"chassis":5,"id":"pcie-root-port4","bus":"pcie.0","addr":"0x4.0x4"}' -device '{"driver":"virtio-net-pci","netdev":"hostnet2","id":"virtio-net-pci2","mac":"00:52:68:26:31:04","bus":"pcie-root-port4"}' -device ide-hd,drive=drive-data-disk,id=system-disk,logical_block_size=512,physical_block_size=512,min_io_size=512,opt_io_size=512,discard_granularity=512,ver=fuxc-ver,bus=ide.0,unit=0 -device '{"driver":"pcie-root-port","port":6,"chassis":6,"id":"pcie-root-port5","bus":"pcie.0","addr":"0x4.0x5"}' -device '{"driver":"ich9-usb-uhci6","id":"uhci6","bus":"pcie-root-port5"}' -device '{"driver":"usb-kbd","id":"kdb0","bus":"uhci6.0"}' -device '{"driver":"pcie-root-port","port":7,"chassis":7,"id":"pcie-root-port6","bus":"pcie.0","addr":"0x4.0x6"}' -device '{"driver":"ich9-usb-uhci5","id":"uhci5","bus":"pcie-root-port6"}' -device '{"driver":"usb-mouse","id":"mouse0","bus":"uhci5.0"}' -device '{"driver":"pcie-root-port","port":8,"chassis":8,"id":"pcie-root-port7","bus":"pcie.0","addr":"0x4.0x7"}' -device '{"driver":"qemu-xhci","id":"xhci","bus":"pcie-root-port7"}' -device '{"driver":"pcie-root-port","port":9,"id":"pcie-root-port8","multifunction":true,"bus":"pcie.0","addr":"0x10"}' -device '{"driver":"usb-ehci","id":"ehci","bus":"pcie-root-port8"}' -device '{"driver":"pcie-root-port","chassis":10,"port":10,"id":"pcie-root-port9","bus":"pcie.0","addr":"0x10.0x1"}' -device '{"driver":"piix3-usb-uhci","id":"usb","bus":"pcie-root-port9"}' -device '{"driver":"pcie-root-port","port":11,"chassis":11,"id":"pcie-root-port10","bus":"pcie.0","addr":"0x10.0x2"}' -device '{"driver":"ich9-usb-uhci3","id":"uhci","bus":"pcie-root-port10"}' -device '{"driver":"usb-storage","drive":"drive-usb-0","id":"usb-0","removable":true,"bus":"uhci.0","port":"1"}' -device '{"driver":"pcie-root-port","port":12,"chassis":12,"id":"pcie-root-port11","bus":"pcie.0","addr":"0x10.0x3"}' -device '{"driver":"pcie-root-port","port":13,"chassis":13,"id":"pcie-root-port12","bus":"pcie.0","addr":"0x10.0x4"}' -device '{"driver":"ich9-usb-ehci1","id":"ehci1","bus":"pcie-root-port11"}' -device '{"driver":"usb-storage","drive":"drive-usb-1","id":"usb-1","removable":true,"bus":"ehci.0","port":"1"}' -device '{"driver":"qemu-xhci","id":"xhci1","bus":"pcie-root-port12"}' -device '{"driver":"usb-storage","drive":"drive-usb-2","id":"usb-2","removable":true,"bus":"xhci.0","port":"1"}' -device '{"driver":"pcie-root-port","port":14,"chassis":14,"id":"pcie-root-port13","bus":"pcie.0","addr":"0x10.0x5"}' -device '{"driver":"virtio-rng-pci","id":"rng0","bus":"pcie-root-port13","rng":"objrng0"}' -device '{"driver":"pcie-root-port","port":15,"chassis":15,"id":"pcie-root-port14","bus":"pcie.0","addr":"0x10.0x6"}' -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pcie-root-port14"}' -device isa-debugcon,chardev=seabioslog_id,iobase=0x402 -device '{"driver":"pcie-root-port","port":16,"chassis":16,"id":"pcie-root-port15","bus":"pcie.0","addr":"0x10.0x7"}' -device '{"driver":"pcie-pci-bridge","id":"pci.1","bus":"pcie-root-port15","addr":"0x0"}' -device '{"driver":"i6300esb","id":"wdt0","bus":"pci.1","addr":"0x1"}' -device '{"driver":"VGA","id":"video0","vgamem_mb":16,"bus":"pcie.0"}' -device '{"driver":"pcie-root-port","port":17,"chassis":17,"id":"pcie-root-port16","bus":"pcie.0","addr":"0x11"}' -device '{"driver":"vhost-vsock-pci","id":"vhost_vsock0","guest-cid":3,"bus":"pcie-root-port16"}' -device '{"driver":"piix3-usb-uhci","id":"piix3-usb-uhci0","bus":"pci.1","addr":"0x4"}' -device '{"driver":"pcie-root-port","port":18,"chassis":19,"id":"pcie-root-port18","bus":"pcie.0","addr":"0x12"}' -device '{"driver":"nec-usb-xhci","p2":4,"p3":4,"id":"nex-usb-xhci0","bus":"pcie-root-port18","addr":"0x12.0x1"}' -device '{"driver":"intel-iommu","intremap":"off","caching-mode":true,"device-iotlb":true}' -audiodev '{"id":"audio1","driver":"none"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/a1.iso","node-name":"drive_sys1"}' -blockdev '{"driver":"raw","node-name":"drive-ide0-1-0","file":"drive_sys1"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/a2.iso","node-name":"drive_sys2"}' -blockdev '{"driver":"raw","node-name":"drive-ide0-1-1","file":"drive_sys2"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/rhel920-64-virtio-scsi-ovmf.qcow2","node-name":"drive_sys3"}' -blockdev '{"driver":"qcow2","node-name":"drive-virtio-disk0","file":"drive_sys3"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/d2.qcow2","node-name":"drive_sys4"}' -blockdev '{"driver":"qcow2","node-name":"drive-scsi-disk","file":"drive_sys4"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/d3.raw","node-name":"drive_sys5"}' -blockdev '{"driver":"raw","node-name":"drive-data-disk","file":"drive_sys5"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/d4.qcow2","node-name":"drive_sys6"}' -blockdev '{"driver":"qcow2","node-name":"usb-disk1","file":"drive_sys6"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/d5.qcow2","node-name":"drive_sys7"}' -blockdev '{"driver":"qcow2","node-name":"usb-disk2","file":"drive_sys7"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/d6.qcow2","node-name":"drive_sys8"}' -blockdev '{"driver":"qcow2","node-name":"usb-disk3","file":"drive_sys8"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/d7.qcow2","node-name":"drive_sys9"}' -blockdev '{"driver":"qcow2","node-name":"drive-usb-0","file":"drive_sys9"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/d8.qcow2","node-name":"drive_sys10"}' -blockdev '{"driver":"qcow2","node-name":"drive-usb-1","file":"drive_sys10"}' -blockdev '{"driver":"file","cache":{"direct":true,"no-flush":false},"filename":"/home/rhel920/d9.qcow2","node-name":"drive_sys11"}' -blockdev '{"driver":"qcow2","node-name":"drive-usb-2","file":"drive_sys11"}' -blockdev '{"node-name":"file_ovmf_code","driver":"file","filename":"/usr/share/OVMF/OVMF_CODE.secboot.fd","auto-read-only":true,"discard":"unmap"}' -blockdev '{"node-name":"drive_ovmf_code","driver":"raw","read-only":true,"file":"file_ovmf_code"}' -blockdev '{"node-name":"file_ovmf_vars","driver":"file","filename":"/home/rhel920/rhel920-64-virtio-scsi-ovmf.qcow2_VARS.fd","auto-read-only":true,"discard":"unmap"}' -blockdev '{"node-name":"drive_ovmf_vars","driver":"raw","read-only":false,"file":"file_ovmf_vars"}' -object '{"qom-type":"memory-backend-ram","id":"mem-1","prealloc":true,"size":2147483648,"host-nodes":[0],"policy":"bind"}' -object '{"qom-type":"memory-backend-ram","id":"mem-2","prealloc":true,"size":2147483648,"host-nodes":[0],"policy":"bind"}' -object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' -numa node,memdev=mem-1 -numa node,memdev=mem-2 -netdev tap,id=hostnet1,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -netdev tap,id=hostnet2,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -m 4096,slots=256,maxmem=32G -smp 8,cores=1,threads=1,sockets=8 -vnc :10 -rtc base=utc,clock=host -boot order=cdn,once=c,menu=on,strict=on -enable-kvm  -qmp tcp:0:3333,server,nowait -qmp tcp:0:9999,server=on,wait=off -qmp tcp:0:9888,server=on,wait=off -serial tcp:0:4444,server,nowait -monitor stdio -watchdog-action reset

So, with with the 9.3 scratch build I provided with the fix: 

9.2 -> 9.3 works,
9.3 -> 9.2 reproduces the bug.

This is expected, as the fixed get_pci_config_device() runs in the target/receiving qemu.
So we need to backport the fix for every target/receiving qemu what will receive a migration for a machine-type <= get_pci_config_device.

Please verify which versions of (receiving/target) qemu are affected by this bug so I can provide the backport

> 1.migrate guest from src to dst  - successfully (but need to wait a while and then the vm was activated on dst side)

This waiting is unexpected, this fix should not affect neither migration time nor downtime.
Please verify if this keeps happening so we can deal with this other issue.

IIUC this bug is not a regression.

I have ran some tests, and this bug also reproduces in earlier RHEL versions of qemu:

RHEL8.6 (tag: qemu-kvm-6.2.0-11.module+el8.6.0+18167+43cf40f3.8)
RHEL9.0 (tag: qemu-kvm-6.2.0-11.el9_0.7)
RHEL9.1 (tag: qemu-kvm-7.0.0-13.el9_1.2)
RHEL9.2 (tag: qemu-kvm-7.2.0-14.el9_2.2)

Min Deng, could you please run your tests with the same scenarios as RHEL9.3 qemu, just changing the qemu version?

For reference, I used the qemu cmdline below, just changing the qemu binary:

#####
/home/leo/qemu/bz2215819/qemu-system-x86_64 \
-cpu Skylake-Client \
-machine pc-q35-rhel8.5.0 \
-m 4G \
-hda rhel-guest-image-9.2-20230531.18.x86_64.qcow2 \
-nodefaults \
-enable-kvm \
-serial mon:stdio \
${migrate} \
-device driver=pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 \
-device nec-usb-xhci,id=xhci,bus=pcie-root-port18,addr=0x12.0x1 \
-device driver=usb-mouse,id=mouse0,bus=xhci.0 \
###

Where:
migrate="-incoming tcp:localhost:12345" #receiving qemu
migrate="" #sending qemu

I suppose some other issue/change outside qemu causes the bug to reproduce now, but I understand it was present in earlier versions as well.

For reference, on upstream it reproduces at least since v6.0.0 with the same cmdline above (machine type changed to pc-q35-6.0)

(In reply to Leonardo Bras from comment #15)
> IIUC this bug is not a regression.
> 
> I have ran some tests, and this bug also reproduces in earlier RHEL versions
> of qemu:
> 
> RHEL8.6 (tag: qemu-kvm-6.2.0-11.module+el8.6.0+18167+43cf40f3.8)
> RHEL9.0 (tag: qemu-kvm-6.2.0-11.el9_0.7)
> RHEL9.1 (tag: qemu-kvm-7.0.0-13.el9_1.2)
> RHEL9.2 (tag: qemu-kvm-7.2.0-14.el9_2.2)
> 
> Min Deng, could you please run your tests with the same scenarios as RHEL9.3
> qemu, just changing the qemu version?

Talked with Leonardo, his tests were on the same versions as below.  Stable guest abi actually focuses on different versions (rhel8.6 to rhel9.0,...), anyway, I will share the test results with you once I get it.
rhel8.6 <-> rhel8.6
rhel9.0 <-> rhel9.0
rhel9.1 <-> rhel9.1
rhel9.2 <-> rhel9.2
Thank you!Any issues please let me know.
Min

QE tried it on rhel 9.3(src:qemu-kvm-8.0.0-6.el9.x86_64) host -> rhel 9.3 (dst:qemu-kvm-8.0.0-6.el9.x86_64) host, the issue can be reproduced too.

Cmdline as below:
/usr/libexec/qemu-kvm -name "mouse-vm" -sandbox on -machine pc-q35-rhel8.4.0,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars -nodefaults  -cpu Cascadelake-Server-noTSX,hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt -chardev socket,id=charmonitor,path=/home/tmp1,server=on,wait=off -chardev pty,id=charserial0 -chardev socket,id=charchannel1,path=/home/tmp2,server=on,wait=off -chardev socket,id=seabioslog_id,path=/home/seabios,server=on,wait=off -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -device ahci,id=ahci0,bus=pcie.0,addr=0x3 -device ide-cd,bus=ahci0.0,unit=0,drive=drive-ide0-1-1,id=ide0-1-1 -device pcie-root-port,port=0x10,chassis=1,id=pcie-root-port0,bus=pcie.0,multifunction=on,addr=0x4 -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port0 -device pcie-root-port,port=0x11,chassis=2,id=pcie-root-port1,bus=pcie.0,addr=0x4.0x1 -device scsi-hd,bus=scsi0.0,lun=0,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,lun=1,id=data-disk1 -device pcie-root-port,port=0x12,chassis=3,id=pcie-root-port2,bus=pcie.0,addr=0x4.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pcie-root-port2 -device isa-serial,chardev=charserial0,id=serial0 -device intel-hda,id=sound0,bus=pcie.0,addr=0x7 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device intel-hda,id=sound1,bus=pcie.0,addr=0x8 -device hda-micro,id=sound1-codec0,bus=sound1.0 -device intel-hda,id=sound2,bus=pcie.0,addr=0x9 -device hda-output,id=sound2-codec0,bus=sound2.0,cad=0 -device ich9-intel-hda,id=sound3,bus=pcie.0,addr=0xa -device hda-duplex,id=sound3-codec0,bus=sound3.0,cad=0 -device pvpanic,ioport=1285 -device pcie-root-port,port=0x13,chassis=4,id=pcie-root-port3,bus=pcie.0,addr=0x4.0x3 -device e1000e,netdev=hostnet1,id=virtio-net-pci1,mac=00:52:68:26:31:03,bus=pcie-root-port3 -device pcie-root-port,port=0x14,chassis=5,id=pcie-root-port4,bus=pcie.0,addr=0x4.0x4 -device virtio-net-pci,netdev=hostnet2,id=virtio-net-pci2,mac=00:52:68:26:31:04,bus=pcie-root-port4 -device ide-hd,drive=drive-data-disk,id=system-disk,logical_block_size=512,physical_block_size=512,min_io_size=512,opt_io_size=512,discard_granularity=512,ver=fuxc-ver,bus=ide.0,unit=0 -device pcie-root-port,port=0x15,chassis=6,id=pcie-root-port5,bus=pcie.0,addr=0x4.0x5 -device ich9-usb-uhci6,id=uhci6,bus=pcie-root-port5 -device usb-kbd,id=kdb0,bus=uhci6.0 -device pcie-root-port,port=0x16,chassis=7,id=pcie-root-port6,bus=pcie.0,addr=0x4.0x6 -device ich9-usb-uhci5,id=uhci5,bus=pcie-root-port6 -device usb-mouse,id=mouse0,bus=uhci5.0 -device pcie-root-port,port=0x17,chassis=8,id=pcie-root-port7,bus=pcie.0,addr=0x4.0x7 -device qemu-xhci,id=xhci,bus=pcie-root-port7 -device pcie-root-port,port=0x18,chassis=9,id=pcie-root-port8,bus=pcie.0,multifunction=on,addr=0x10 -device usb-ehci,id=ehci,bus=pcie-root-port8 -device pcie-root-port,port=0x19,chassis=10,id=pcie-root-port9,bus=pcie.0,addr=0x10.0x1 -device piix3-usb-uhci,id=usb,bus=pcie-root-port9 -device pcie-root-port,port=0x20,chassis=11,id=pcie-root-port10,bus=pcie.0,addr=0x10.0x2 -device ich9-usb-uhci3,id=uhci,bus=pcie-root-port10 -device usb-storage,drive=drive-usb-0,id=usb-0,removable=on,bus=uhci.0,port=1 -device pcie-root-port,port=0x21,chassis=12,id=pcie-root-port11,bus=pcie.0,addr=0x10.0x3 -device pcie-root-port,port=0x22,chassis=13,id=pcie-root-port12,bus=pcie.0,addr=0x10.0x4 -device ich9-usb-ehci1,id=ehci1,bus=pcie-root-port11 -device usb-storage,drive=drive-usb-1,id=usb-1,removable=on,bus=ehci1.0,port=1 -device qemu-xhci,id=xhci1,bus=pcie-root-port12 -device usb-storage,drive=drive-usb-2,id=usb-2,removable=on,bus=xhci1.0,port=1 -device pcie-root-port,port=0x23,chassis=14,id=pcie-root-port13,bus=pcie.0,addr=0x10.0x5 -device virtio-rng-pci,id=rng0,bus=pcie-root-port13,rng=objrng0 -device pcie-root-port,port=0x24,chassis=15,id=pcie-root-port14,bus=pcie.0,addr=0x10.0x6 -device virtio-balloon-pci,id=balloon0,bus=pcie-root-port14 -device isa-debugcon,chardev=seabioslog_id,iobase=0x402 -device pcie-root-port,port=0x25,chassis=16,id=pcie-root-port15,bus=pcie.0,addr=0x10.0x7 -device pcie-pci-bridge,id=pci.1,bus=pcie-root-port15,addr=0x0 -device i6300esb,id=wdt0,bus=pci.1,addr=0x1 -device VGA,id=video0,vgamem_mb=16,bus=pcie.0 -device pcie-root-port,port=0x26,chassis=17,id=pcie-root-port16,bus=pcie.0,addr=0x11 -device vhost-vsock-pci,id=vhost_vsock0,guest-cid=3,bus=pcie-root-port16 -device piix3-usb-uhci,id=piix3-usb-uhci0,bus=pci.1,addr=0x4 -device pcie-root-port,port=18,chassis=19,id=pcie-root-port18,bus=pcie.0,addr=0x12 -device nec-usb-xhci,id=nex-usb-xhci0,bus=pcie-root-port18,addr=0x12.0x1 -audiodev id=audio1,driver=none -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/a1.iso,node-name=drive_sys1 -blockdev driver=raw,node-name=drive-ide0-1-0,file=drive_sys1 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/a2.iso,node-name=drive_sys2 -blockdev driver=raw,node-name=drive-ide0-1-1,file=drive_sys2 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/rhel920-64-virtio-scsi-ovmf.qcow2,node-name=drive_sys3 -blockdev driver=qcow2,node-name=drive-virtio-disk0,file=drive_sys3 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/d2.qcow2,node-name=drive_sys4 -blockdev driver=qcow2,node-name=drive-scsi-disk,file=drive_sys4 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/d3.raw,node-name=drive_sys5 -blockdev driver=raw,node-name=drive-data-disk,file=drive_sys5 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/d4.qcow2,node-name=drive_sys6 -blockdev driver=qcow2,node-name=usb-disk1,file=drive_sys6 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/d5.qcow2,node-name=drive_sys7 -blockdev driver=qcow2,node-name=usb-disk2,file=drive_sys7 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/d6.qcow2,node-name=drive_sys8 -blockdev driver=qcow2,node-name=usb-disk3,file=drive_sys8 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/d7.qcow2,node-name=drive_sys9 -blockdev driver=qcow2,node-name=drive-usb-0,file=drive_sys9 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/d8.qcow2,node-name=drive_sys10 -blockdev driver=qcow2,node-name=drive-usb-1,file=drive_sys10 -blockdev driver=file,cache.direct=on,cache.no-flush=off,filename=/home/rhel920/d9.qcow2,node-name=drive_sys11 -blockdev driver=qcow2,node-name=drive-usb-2,file=drive_sys11 -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/rhel920/rhel920-64-virtio-scsi-ovmf.qcow2_VARS.fd,auto-read-only=on,discard=unmap -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars -object memory-backend-ram,policy=bind,id=mem-1,size=2048M,prealloc=yes,host-nodes=0 -object memory-backend-ram,policy=bind,id=mem-2,size=2048M,prealloc=yes,host-nodes=0 -object rng-random,filename=/dev/urandom,id=objrng0 -numa node,memdev=mem-1 -numa node,memdev=mem-2 -netdev tap,id=hostnet1,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -netdev tap,id=hostnet2,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -m 4096,slots=256,maxmem=32G -smp 8,cores=1,threads=1,sockets=8 -vnc :10 -rtc base=utc,clock=host -boot order=cdn,once=c,menu=on,strict=on -enable-kvm  -qmp tcp:0:3333,server,nowait -qmp tcp:0:9999,server=on,wait=off -qmp tcp:0:9888,server=on,wait=off -serial tcp:0:8181,server,nowait -monitor stdio -watchdog-action reset -incoming defer
char device redirected to /dev/pts/2 (label charserial0)
qemu-kvm: warning: Machine type 'pc-q35-rhel8.4.0' is deprecated: machine types for previous major releases are deprecated
QEMU 8.0.0 monitor - type 'help' for more information



(qemu) 
(qemu) migrate_incoming tcp:[::]:4000
(qemu) 

Actual results:
(qemu) qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
qemu-kvm: Failed to load PCIDevice:config
qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:12.0/pcie-root-port'
qemu-kvm: load of migration failed: Invalid argument

(In reply to Leonardo Bras from comment #16)
> I suppose some other issue/change outside qemu causes the bug to reproduce
> now

I think this is the most confusing part - if this is a bug that existed since rhel8.5, then why did we only hit it until now?  IIUC we should have hit it since 8.6 as long as we have any machine type set ACPI_PM_PROP_ACPI_PCIHP_BRIDGE=off (of pc_rhel_8_5_compat).  

And as far as I read the comments, it's not a guest ABI break, it's just a migration bug, so it happens too even if we migrate between same version QEMUs.  That's even more weird, because we never hit it in migration tests since RHEL8.6.  Copy Xiaohui too, just in case she has any comment.

Min, do you know is this bug only exposed in a new test added in RHEL9.3?  It should at least also hit in RHEL9.2 guest abi test, why it wasn't hit before?

(In reply to Peter Xu from comment #19)
> (In reply to Leonardo Bras from comment #16)
> > I suppose some other issue/change outside qemu causes the bug to reproduce
> > now
> 
> I think this is the most confusing part - if this is a bug that existed
> since rhel8.5, then why did we only hit it until now?  IIUC we should have
> hit it since 8.6 as long as we have any machine type set
> ACPI_PM_PROP_ACPI_PCIHP_BRIDGE=off (of pc_rhel_8_5_compat).  

It's not that simple to reproduce.

It would need the guest to offline the pci device connected to the pci bus, and cause qemu to hot-unplug it.
Then we would hit the bug.

For some reason the guest current is offlining this particuakar pci device, and that causes the bug to reproduce.


> 
> And as far as I read the comments, it's not a guest ABI break, it's just a
> migration bug, so it happens too even if we migrate between same version
> QEMUs.  

Correct

(In reply to Leonardo Bras from comment #21)
> It would need the guest to offline the pci device connected to the pci bus,
> and cause qemu to hot-unplug it.
> Then we would hit the bug.

Ah... I never see anywhere in this bz mentioning the offline procedure (e.g. comment 0 doesn't seem to have it).  How did you know this?

It definitely should be mentioned somewhere!  That explains.

> 
> For some reason the guest current is offlining this particuakar pci device,
> and that causes the bug to reproduce.

Now I can finally understand why that bit is cleared..  would you mind mentioning the offline procedure also in the upstream patch if there's a new version?  I don't see that mentioned either at least in latest v2:

https://lore.kernel.org/r/20230706045546.593605-3-leobras@redhat.com

I had a feeling that it is also the major thing that confused Michael.  We need Michael's attention to merge/ack that before July 11th for upstream - we still have 5 days.

https://wiki.qemu.org/Planning/8.1

(In reply to Peter Xu from comment #22)
> (In reply to Leonardo Bras from comment #21)
> > It would need the guest to offline the pci device connected to the pci bus,
> > and cause qemu to hot-unplug it.
> > Then we would hit the bug.
> 
> Ah... I never see anywhere in this bz mentioning the offline procedure (e.g.
> comment 0 doesn't seem to have it).  How did you know this?
> 
> It definitely should be mentioned somewhere!  That explains.
> 
> > 
> > For some reason the guest current is offlining this particuakar pci device,
> > and that causes the bug to reproduce.
> 
> Now I can finally understand why that bit is cleared..  would you mind
> mentioning the offline procedure also in the upstream patch if there's a new
> version?  I don't see that mentioned either at least in latest v2:
> 
> https://lore.kernel.org/r/20230706045546.593605-3-leobras@redhat.com

Oh, I totally missed updating the commit msg  on this.

I need to update this part:
In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(),
which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI
hotplug for the guest. After a while the guest will deal with this hotplug
and qemu will clear the above bit.

to:

I need to update this part:
In this scenario, hotplug_handler_plug() calls pcie_cap_slot_plug_cb(),
which sets dev->config byte 0x6e with bit PCI_EXP_SLTSTA_PDS to signal PCI
hotplug for the guest. After a while, if the guest powers down the device
qemu will clear the above bit.



> 
> I had a feeling that it is also the major thing that confused Michael.  We
> need Michael's attention to merge/ack that before July 11th for upstream -
> we still have 5 days.
> 
> https://wiki.qemu.org/Planning/8.1

Thanks for pointing that out!
I will save this page for future version planning :)

More upstream testing looking for the bug:

qemu v5.0.1 reproduces
qemu v4.0.1 reproduces
qemu v3.0.1 reproduces
qemu v2.12.1 reproduces

Older qemu require python2, and I am testing in RHEL9

Seems to be a very old bug, that haven't bother anyone until now.

(In reply to Peter Xu from comment #19)
> (In reply to Leonardo Bras from comment #16)
> > I suppose some other issue/change outside qemu causes the bug to reproduce
> > now
> 
> I think this is the most confusing part - if this is a bug that existed
> since rhel8.5, then why did we only hit it until now?  IIUC we should have
> hit it since 8.6 as long as we have any machine type set
> ACPI_PM_PROP_ACPI_PCIHP_BRIDGE=off (of pc_rhel_8_5_compat).  
> 
> And as far as I read the comments, it's not a guest ABI break, it's just a
> migration bug, so it happens too even if we migrate between same version
> QEMUs.  That's even more weird, because we never hit it in migration tests
> since RHEL8.6.  Copy Xiaohui too, just in case she has any comment.
> 
> Min, do you know is this bug only exposed in a new test added in RHEL9.3? 
> It should at least also hit in RHEL9.2 guest abi test, why it wasn't hit
> before?
Hi Peter,
Good question, I've expanded test cases in stable guest abi test plan this time,that might be the reason why we find it. As peter said it's a migration bug so I will return the QA contact.
Thank you
Min

Hi all,
I can reproduce this bug when migrate VM on the latest rhel 9.3 (qemu-kvm-8.0.0-6.el9.x86_64) with pc-q35-rhel8.5.0 machine type:

1. qemu cmd:
-device '{"driver":"pcie-root-port","id":"extra_root0","multifunction":true,"bus":"pcie.0","addr":"0x3","chassis":21}' \
-device '{"driver":"nec-usb-xhci","id":"usb1","bus":"extra_root0","addr":"0x3.0x1"}' \

2. Qemu on dst host automatically quit with error:
(qemu) 2023-07-10T06:02:15.724905Z qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19
2023-07-10T06:02:15.724944Z qemu-kvm: Failed to load PCIDevice:config
2023-07-10T06:02:15.724948Z qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
2023-07-10T06:02:15.724953Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:03.0/pcie-root-port'
2023-07-10T06:02:15.728480Z qemu-kvm: load of migration failed: Invalid argument


But when change addr of nec-usb-xhci to be 0x0 like below qemu cmd, or use rhel9 machine type, won't hit this bug during migration:
************************************************************************************************************************
-device '{"driver":"pcie-root-port","id":"extra_root0","multifunction":true,"bus":"pcie.0","addr":"0x3","chassis":21}' \
-device '{"driver":"nec-usb-xhci","id":"usb1","bus":"extra_root0","addr":"0x0"}' \



Jing, please help have a look from the Q35 feature, and assign it to the relevant person if need.

Note:
No matter it's nec-usb-xhci or virtio-net-pci, etc, as long as it's under the pcie-root-port and the addr is not 0x0, you can reproduce this bug with machine type lower than rhel 8.6.0

Hi Jing, Wei, Is this still a TestBlocker or can we remove that?

(In reply to Nitesh Narayan Lal from comment #29)
> Hi Jing, Wei, Is this still a TestBlocker or can we remove that?

Hi Nitesh

Based on comment 27, I think we can remove TestBlocker 

Please correct me if need

Thanks
Jing

The issue can be reproduced from libvirt side with the latest machine type (RHEL 9.2.0).Just update the commands here for reference. I converted the command as the following.
/usr/libexec/qemu-kvm \
	-name guest=rhel9 \
	-machine pc-q35-rhel9.2.0,usb=off,smm=on,dump-guest-core=off \
	-accel kvm \
	-cpu Broadwell-noTSX-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on \
	-global driver=cfi.pflash01,property=secure,value=on \
	-m size=2097152k,slots=8,maxmem=6195200k \
	-overcommit mem-lock=off \
	-smp 16,sockets=16,cores=1,threads=1 \
	-object '{"qom-type":"memory-backend-ram","id":"ram-node0","share":true,"size":1073741824}' \
	-numa node,nodeid=0,cpus=0,cpus=2,cpus=4,cpus=6,cpus=8,cpus=10,cpus=12,cpus=14,memdev=ram-node0 \
	-object '{"qom-type":"memory-backend-ram","id":"ram-node1","size":1073741824}' \
	-numa node,nodeid=1,cpus=1,cpus=3,cpus=5,cpus=7,cpus=9,cpus=11,cpus=13,cpus=15,memdev=ram-node1 \
	-numa dist,src=0,dst=0,val=10 \
	-numa dist,src=0,dst=1,val=19 \
	-numa dist,src=1,dst=0,val=19 \
	-numa dist,src=1,dst=1,val=10 \
	-uuid c3b3d47e-0e41-43f2-8455-8061f2f32496 \
	-no-user-config \
	-nodefaults \
	-rtc base=utc,driftfix=slew \
	-global kvm-pit.lost_tick_policy=delay \
	-no-hpet \
	-no-shutdown \
	-global ICH9-LPC.disable_s3=1 \
	-global ICH9-LPC.disable_s4=1 \
	-boot strict=on \
	-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
	-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
	-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
	-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
	-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
	-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
	-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
	-device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \
	-device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \
	-device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \
	-device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \
	-device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \
	-device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \
	-device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \
	-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.2","addr":"0x0"}' \
	-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"}' \
	-blockdev '{"driver":"file","filename":"rhel920-64-virtio-scsi-ovmf.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
	-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
	-device '{"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1}' \
	-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
	-audiodev '{"id":"audio1","driver":"none"}' \
	-vnc 0.0.0.0:0,audiodev=audio1 \
	-device '{"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \
	-incoming defer \
	-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.5","addr":"0x0"}' \
	-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
	-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.6","addr":"0x0"}' \
	-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
	-msg timestamp=on \
	-monitor stdio 

Actual results,
Error happened while doing backward migration
[root@dell-per740-81 rhel920]# sh boot-libvirt-dst.sh 
QEMU 7.2.0 monitor - type 'help' for more information
(qemu) migrate_incoming tcp:[::]:4000
(qemu) 2023-07-11T13:36:17.920402Z qemu-kvm: get_pci_config_device: Bad config data: i=0x10a read: 40 device: 0 cmask: ff wmask: 0 w1cmask:0
2023-07-11T13:36:17.920453Z qemu-kvm: Failed to load PCIDevice:config
2023-07-11T13:36:17.920460Z qemu-kvm: Failed to load pcie-root-port:parent_obj.parent_obj.parent_obj
2023-07-11T13:36:17.920481Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0/pcie-root-port'
2023-07-11T13:36:17.921849Z qemu-kvm: load of migration failed: Invalid argument

(In reply to Yiqian Wei from comment #33)
> Can reproduce this bug with simple command line from RHEL9.3.0 host
> migration to RHEL9.2.0 host.
> 
> 1.Host version:
> RHEL9.3.0:
> qemu-kvm-8.0.0-7.el9.x86_64
> kernel-5.14.0-325.el9.x86_64
> 
> RHEL9.2.0:
> qemu-kvm-7.2.0-14.el9_2.3.x86_64
> kernel-5.14.0-284.18.1.el9_2.x86_64
> 
> 2.Simple command line in src and dst end
> In src end:
> # /usr/libexec/qemu-kvm  -name guest=rhel9 -machine pc-q35-rhel9.2.0
> -monitor stdio -vnc :1 -cpu Broadwell-noTSX-IBRS,enforce
> QEMU 8.0.0 monitor - type 'help' for more information
> (qemu) 
> 
> In dst end:
> # /usr/libexec/qemu-kvm  -name guest=rhel9 -machine pc-q35-rhel9.2.0
> -monitor stdio -vnc :1 -cpu Broadwell-noTSX-IBRS,enforce -incoming defer 
> QEMU 7.2.0 monitor - type 'help' for more information
> (qemu) migrate_incoming tcp::1234
> (qemu) 
> 
> 3.migration RHEL9.3.0 to RHEL9.2.0
> (qemu) migrate -d tcp:10.73.212.86:1234
> 
> Test results: Migration failed in dst end
> (qemu) qemu-kvm: get_pci_config_device: Bad config data: i=0x10a read: 40
> device: 0 cmask: ff wmask: 0 w1cmask:0
> qemu-kvm: Failed to load PCIDevice:config
> qemu-kvm: Failed to load e1000e:parent_obj
> qemu-kvm: error while loading state for instance 0x0 of device
> '0000:00:02.0/e1000e'
> qemu-kvm: load of migration failed: Invalid argument

Hello developers,
It looks like it's new issue cause the similar result, can you help to confirm it ? I'd like to open a new bug if you agree with me, thank you!
In src(9.3):
In dst(9.2):
Migration started from high qemu-kvm version to low qemu-kvm version, and it can be reproduced from libvirt side. Thank you!

Thanks
Min

(In reply to Min Deng from comment #34)
> 
> (In reply to Yiqian Wei from comment #33)
> > Can reproduce this bug with simple command line from RHEL9.3.0 host
> > migration to RHEL9.2.0 host.
> > 
> > 1.Host version:
> > RHEL9.3.0:
> > qemu-kvm-8.0.0-7.el9.x86_64
> > kernel-5.14.0-325.el9.x86_64
> > 
> > RHEL9.2.0:
> > qemu-kvm-7.2.0-14.el9_2.3.x86_64
> > kernel-5.14.0-284.18.1.el9_2.x86_64
> > 
> > 2.Simple command line in src and dst end
> > In src end:
> > # /usr/libexec/qemu-kvm  -name guest=rhel9 -machine pc-q35-rhel9.2.0
> > -monitor stdio -vnc :1 -cpu Broadwell-noTSX-IBRS,enforce
> > QEMU 8.0.0 monitor - type 'help' for more information
> > (qemu) 
> > 
> > In dst end:
> > # /usr/libexec/qemu-kvm  -name guest=rhel9 -machine pc-q35-rhel9.2.0
> > -monitor stdio -vnc :1 -cpu Broadwell-noTSX-IBRS,enforce -incoming defer 
> > QEMU 7.2.0 monitor - type 'help' for more information
> > (qemu) migrate_incoming tcp::1234
> > (qemu) 
> > 
> > 3.migration RHEL9.3.0 to RHEL9.2.0
> > (qemu) migrate -d tcp:10.73.212.86:1234
> > 
> > Test results: Migration failed in dst end
> >
> > device: 0 cmask: ff wmask: 0 w1cmask:0
> > qemu-kvm: Failed to loa (qemu) qemu-kvm: get_pci_config_device: Bad config data: i=0x10a read: 40d PCIDevice:config
> > qemu-kvm: Failed to load e1000e:parent_obj
> > qemu-kvm: error while loading state for instance 0x0 of device
> > '0000:00:02.0/e1000e'
> > qemu-kvm: load of migration failed: Invalid argument
> 
> Hello developers,
> It looks like it's new issue cause the similar result, can you help to
> confirm it ? I'd like to open a new bug if you agree with me, thank you!

It's a similar, but different bug:
New one: qemu-kvm: get_pci_config_device: Bad config data: i=0x10a read: 40 device: 0 cmask: ff wmask: 0 w1cmask:0
This BZ: qemu-kvm: get_pci_config_device: Bad config data: i=0x6e read: 0 device: 40 cmask: ff wmask: 0 w1cmask:19

The index is different, so this BZ's fix will not fix above issue.

Hi Leonardo,
Thanks for confirming it, reported new one. 
Bug 2223691 - [machine type 9.2]Failed to migrate VM from RHEL 9.3 to RHEL 9.2
Thanks
Min

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

I want to highlight that the issue were not reproduced with machine type higher than 8.6.0. It's only reproduced with machine type which is lower than 8.6.0, at this point, it makes sense ? Feel free to correct me if I were wrong.

Thanks for asking the valid question, Yiqian.  My understanding is that the bug can also be triggered even with correct address but as long as we offline the slot before migrate.  Anyone please correct me otherwise.

PS: we should always have the pci experts in the loop for these questions at least to keep our discussion honest.  Adding Michael / Igor (just in case for opt-in anytime).  And let's do this for every time we found anything could be pci related so they're always aware from the start.

Hello All,
Per comment 47, could you please also help to evaluate if we need to do the back port ? Thank you all.
Thanks 
Min

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368