Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1867048

Summary: After hotpluging the failover VF and then rebooting the vm, qemu core dump
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Yanghang Liu <yanghliu>
Component: qemu-kvmAssignee: Juan Quintela <quintela>
qemu-kvm sub component: Networking QA Contact: Yanghang Liu <yanghliu>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: high CC: aadam, chayang, jinzhao, juzhang, lvivier, ngu, quintela, virt-maint, yalzhang
Version: 8.3Keywords: Regression, Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-12 02:27:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
(gdb)t a a bt full none

Description Yanghang Liu 2020-08-07 09:01:30 UTC
Created attachment 1710765 [details]
(gdb)t a a bt full

Description of problem:
qemu core dump after hotpluging the failover VF and rebooting the vm

Version-Release number of selected component (if applicable):
host:
qemu-kvm-5.0.0-2.module+el8.3.0+7379+0505d6ca.x86_64
4.18.0-229.el8.x86_64
guest:
4.18.0-229.el8.x86_64


How reproducible:
100%

Steps to Reproduce:
1.Create a VF and bind the VF driver to vfio-pci 
echo 1 > /sys/bus/pci/devices/0000\:82\:00.0/sriov_numvfs
echo 0000:82:01.0 > /sys/bus/pci/devices/0000\:82\:01.0/driver/unbind
echo "14e4 16af" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "14e4 16af" > /sys/bus/pci/drivers/vfio-pci/remove_id

2.Set the MAC address of the VF
ip link set enp130s0f0 vf 0  mac 22:2b:62:bb:a9:82

3.start a vm with a failover virtio netdev
/usr/libexec/qemu-kvm -name rhel83 \
-M q35 \
-enable-kvm \
-monitor stdio \
-nodefaults \
-m 4G \
-boot menu=on \
-cpu Haswell-noTSX-IBRS \
-device pcie-root-port,id=root.1,chassis=1,addr=0x2.0,multifunction=on \
-device pcie-root-port,id=root.2,chassis=2,addr=0x2.1 \
-device pcie-root-port,id=root.3,chassis=3,addr=0x2.2 \
-device pcie-root-port,id=root.4,chassis=4,addr=0x2.3 \
-device pcie-root-port,id=root.5,chassis=5,addr=0x2.4 \
-device pcie-root-port,id=root.6,chassis=6,addr=0x2.5 \
-device pcie-root-port,id=root.7,chassis=7,addr=0x2.6 \
-device pcie-root-port,id=root.8,chassis=8,addr=0x2.7 \
-smp 8 \
-qmp tcp:0:5555,server,nowait \
-blockdev node-name=back_image,driver=file,cache.direct=on,cache.no-flush=off,filename=/nfsmount/migra_test/229.qcow2,aio=threads \
-blockdev node-name=drive-virtio-disk0,driver=qcow2,cache.direct=on,cache.no-flush=off,file=back_image \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=disk0,bus=root.1 \
-device VGA,id=video1,bus=root.2  \
-vnc :0 \
-netdev tap,id=hostnet0,vhost=on \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=22:2b:62:bb:a9:82,bus=root.3,failover=on \

output:
(qemu) qemu-kvm: warning: Primary device not found
Virtio-net failover will not work. Make sure primary device has parameter failover_pair_id=<virtio-net-id>

4.hotplug a failover vf into vm
hmp:
(qemu) device_add vfio-pci,host=0000:82:01.0,id=hostdev0,bus=root.4,failover_pair_id=net0

or

qmp:
{"execute":"device_add","arguments":{"driver":"vfio-pci","host":"82:01.0","id":"vf","bus":"root.4","failover_pair_id":"net0"}} 


5.check the failover vf info in vm
# dmesg
[  114.848136] pcieport 0000:00:02.3: Slot(0-3): Attention button pressed
[  114.849189] pcieport 0000:00:02.3: Slot(0-3) Powering on due to button press
[  114.850411] pcieport 0000:00:02.3: Slot(0-3): Card present
[  114.851289] pcieport 0000:00:02.3: Slot(0-3): Link Up
[  114.976885] pci 0000:04:00.0: [14e4:16af] type 00 class 0x020000
[  114.978363] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x00007fff 64bit pref]
[  114.979727] pci 0000:04:00.0: reg 0x20: [mem 0x00000000-0x00001fff 64bit pref]
[  114.980971] pci 0000:04:00.0: enabling Extended Tags
[  114.982559] pci 0000:04:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:02.3 (capable of 32.000 Gb/s with 5 GT/s x8 link)
[  114.987600] pci 0000:04:00.0: BAR 0: assigned [mem 0xfc800000-0xfc807fff 64bit pref]
[  114.989680] pci 0000:04:00.0: BAR 4: assigned [mem 0xfc808000-0xfc809fff 64bit pref]
[  114.991718] pcieport 0000:00:02.3: PCI bridge to [bus 04]
[  114.993358] pcieport 0000:00:02.3:   bridge window [io  0x4000-0x4fff]
[  114.997631] pcieport 0000:00:02.3:   bridge window [mem 0xfe200000-0xfe3fffff]
[  115.000942] pcieport 0000:00:02.3:   bridge window [mem 0xfc800000-0xfc9fffff 64bit pref]
[  115.084484] bnx2x 0000:04:00.0: msix capability found
[  115.088529] bnx2x 0000:04:00.0: enabling device (0000 -> 0002)
[  115.325191] bnx2x 0000:04:00.0 eth0: using MSI-X  IRQs: fp[0] 38 ... fp[7] 45
[  115.327130] bnx2x 0000:04:00.0 eth0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
[  116.473020] virtio_net virtio1 enp3s0: failover primary slave:eth0 registered
[  116.475058] bnx2x 0000:04:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s x1 link at 0000:00:02.3 (capable of 32.000 Gb/s with 5 GT/s x8 link)


# ifconfig 
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.33.130  netmask 255.255.254.0  broadcast 10.73.33.255
        inet6 fe80::cc14:a7d3:815c:f9f1  prefixlen 64  scopeid 0x20<link>
        inet6 2620:52:0:4920:6ed3:cc68:dbaf:5f88  prefixlen 64  scopeid 0x0<global>
        ether 22:2b:62:bb:a9:82  txqueuelen 1000  (Ethernet)
        RX packets 345  bytes 39735 (38.8 KiB)
        RX errors 0  dropped 69  overruns 0  frame 0
        TX packets 218  bytes 93399 (91.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0nsby: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.33.130  netmask 255.255.254.0  broadcast 10.73.33.255
        inet6 fe80::53cd:a8d1:685e:39fe  prefixlen 64  scopeid 0x20<link>
        ether 22:2b:62:bb:a9:82  txqueuelen 1000  (Ethernet)
        RX packets 331  bytes 37304 (36.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 204  bytes 91735 (89.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.33.204  netmask 255.255.254.0  broadcast 10.73.33.255
        inet6 fe80::9bce:ac4:7a18:18b8  prefixlen 64  scopeid 0x20<link>
        ether 22:2b:62:bb:a9:82  txqueuelen 1000  (Ethernet)
        RX packets 14  bytes 2431 (2.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 1664 (1.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfc800000-fc807fff  


6.reboot the vm

output:
(qemu) Unexpected error in object_property_try_add() at /builddir/build/BUILD/qemu-5.0.0/qom/object.c:1167:
qemu-kvm: attempt to add duplicate property 'hostdev0' to object (type 'container')



Actual results:
qemu core dump

Expected results:
The failover vf is hotplugged into vm successfully
The vm always works well.

Additional info:
(1)Use qemu-kvm-4.2.0-31.module+el8.3.0+7437+4bb96e0d.x86_64 to repeat steps 1 to 5, this problem cannot be reproduced.
(2)The backtrace info about qemu core dump is in the attachment.
(3)This bug can be reproduced with XL710,Broadcom BCM57810,82599ES,MT27800

Comment 1 Yanghang Liu 2020-08-07 12:01:35 UTC
It seems that this bug is a duplicate bug of Bug 1852354 - Hotplug memory with the same id to previous one, qemu got coredump

Comment 2 Yanghang Liu 2020-09-11 15:43:26 UTC
This bug can still be reproduced in qemu-kvm-5.1.0-5.module+el8.3.0+7975+b80d25f1.x86_64  with the following error info:

(qemu) Unexpected error in object_property_try_add() at /builddir/build/BUILD/qemu-5.1.0/qom/object.c:1181:
qemu-kvm: attempt to add duplicate property 'vf' to object (type 'container')

Comment 4 Juan Quintela 2020-10-07 11:28:01 UTC
Hi

I have a fix for this pending to sent to the main list.

Comment 10 Laurent Vivier 2021-03-03 08:18:07 UTC
It looks like a duplicate of BZ 1819991

Comment 11 Yanghang Liu 2021-03-04 05:49:23 UTC

(In reply to Laurent Vivier from comment #10)
> It looks like a duplicate of BZ 1819991

Bug 1819991 has been fixed in the qemu-kvm-5.2.0-10.module+el8.4.0+10217+cbdd2152.



I have tested this bug with the following two qemu-kvm versions:

(1) qemu-kvm-5.2.0-9.module+el8.4.0+10182+4161bd91.x86_64

This bug *can* be reproduced. 

The qemu will core dump after hot-plugging the failover VF and then rebooting the vm.

> Unexpected error in object_property_try_add() at ../qom/object.c:1219:
> 2021-03-04T04:07:20.338545Z qemu-kvm: attempt to add duplicate property 'hostdev0' to object (type 'container')
> 2021-03-04 04:07:21.056+0000: shutting down, reason=crashed


(2) qemu-kvm-5.2.0-10.module+el8.4.0+10217+cbdd2152.x86_64

This bug *can not* be reproduced. 

The vm works well after hot-plugging the failover VF and then rebooting the vm.

Comment 12 Juan Quintela 2021-03-11 08:44:04 UTC
hi

As per comment 11 says that the bug is fixed, can we close it?

Comment 13 Yanghang Liu 2021-03-12 02:27:42 UTC
According to comment 10, comment 11 and comment 12 , close this bz.

*** This bug has been marked as a duplicate of bug 1819991 ***