Bug 1708480

Summary: [Q35] No "DEVICE_DELETED" event in qmp after unplug virtio-net-pci device
Product: Red Hat Enterprise Linux 8 Reporter: Peixiu Hou <phou>
Component: qemu-kvmAssignee: Julia Suvorova <jusual>
qemu-kvm sub component: PCI QA Contact: Lei Yang <leiyang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: ailan, jinzhao, jusual, juzhang, leiyang, lmiksik, pezhang, rbalakri, rkhan, virt-maint, ybendito, yiwei, yuhuang
Version: 8.0   
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-2.12.0-98.module+el8.2.0+5698+10a84757 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1791590 (view as bug list) Environment:
Last Closed: 2020-04-28 15:32:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1710667, 1744438, 1791590    

Description Peixiu Hou 2019-05-10 03:37:28 UTC
Description of problem:
No "DEVICE_DELETED" event in qmp after unplug virtio-net-pci device
It cannnot be reproduced with pc machine.

Version-Release number of selected component (if applicable):
Used versions:
kernel-4.18.0-80.20.el8.x86_64
qemu-kvm-3.1.0-22.module+el8.0.1+3032+a09688b9.x86_64
seabios-bin-1.12.0-1.module+el8+2706+3c6581b6.noarch
virtio-win-prewhql-169/171

How reproducible:
100% on fast train, 20% on slow train

Steps to Reproduce:
steps:
1. boot guest up with virtio-net-pci device:
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/win7-32-sp1-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \
    -netdev tap,script=/etc/qemu-ifup,downscript=no,id=hostnet1,vhost=on \
    -device virtio-net-pci,netdev=hostnet1,id=net1,mac=00:52:20:76:1a:4d,bus=pcie.0-root-port-5,addr=0x0 \
    -m 24576  \
    -smp 32,maxcpus=32,cores=16,threads=1,sockets=2  \
    -cpu 'EPYC',hv_stimer,hv_synic,hv_vpindex,hv_reset,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv-tlbflush,+kvm_pv_unhalt \
    -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso \
    -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
    -drive id=drive_virtio,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/virtio-win-prewhql-0.1-171.iso \
    -device ide-cd,id=virtio,drive=drive_virtio,bus=ide.1,unit=0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :5  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp::4445,server,nowait \
    -device pcie-root-port,id=pcie_extra_root_port_0,slot=6,chassis=6,addr=0x6,bus=pcie.0 \


2. From qmp monitor, unplug the virtio-net-pci device.
telnet host_ip 4445
{ 'execute': 'qmp_capabilities' }
{ 'execute': 'device_del', 'arguments': {'id': 'net1' }}

3. check the "event": "DEVICE_DELETED" logs, only return "/machine/peripheral/net1/virtio-backend" be deleted info, as follow: 
{"timestamp": {"seconds": 1554088482, "microseconds": 474877}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/net1/virtio-backend"}}
no return {"device": "net0", "path": "/machine/peripheral/net1"} be deleted info, it should return follow as well:
{"timestamp": {"seconds": 1511509084, "microseconds": 162214}, "event": "DEVICE_DELETED", "data": {"device": "net1", "path": "/machine/peripheral/net0"}}

Actual results:
no return {"device": "net0", "path": "/machine/peripheral/net1"}

Expected results:
return {"device": "net0", "path": "/machine/peripheral/net1"}

Additional info:
It cannnot be reproduced with pc machine.

Comment 3 Lei Yang 2019-05-15 07:16:11 UTC
Vfio-pci nic hit this bug. 

Used versions:
qemu-kvm-4.0.0-0.module+el8.1.0+3169+3c501422.x86_64
kernel-4.18.0-86.el8.x86_64
virtio-win-1.9.7-3.el8.noarch

Steps to Reproduce:
1.Boot win2019 guest with vfio-pci nic.
Qemu cli:
/usr/libexec/qemu-kvm -name Win2019 \
-M q35,kernel-irqchip=split -m 4G \
-nodefaults \
-cpu Haswell-noTSX \
-device intel-iommu,intremap=on,caching-mode=on \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,chassis=1 \
-device pcie-root-port,id=root.2,chassis=2 \
-device pcie-root-port,id=root.3,chassis=3 \
-device pcie-root-port,id=root.4,chassis=4 \
-device pcie-root-port,id=root.5,chassis=5 \
-blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/win2019.qcow2,node-name=my_file \
-drive id=drive_cd1,if=none,snapshot=off,aio=native,cache=none,media=cdrom,file=/home/en_windows_server_2019_updated_march_2019_x64_dvd_2ae967ab.iso \
-blockdev driver=qcow2,node-name=my,file=my_file \
-device virtio-blk-pci,drive=my,id=virtio-blk0,bus=root.1 \
-device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
-vnc :1 \
-vga qxl \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \
-usb -device usb-tablet \
-boot menu=on \
-device vfio-pci,host=0000:04:00.0,id=pf-1,bus=root.3 \

2.From qmp monitor,unplug this nic.
#telnet 10.73.73.73 5554
{"execute": "qmp_capabilities"}
{"execute":"device_del","arguments":{"id":"pf-1"}

Comment 6 Amnon Ilan 2019-06-27 08:16:28 UTC
Maybe related comment from Vadim here: https://bugzilla.redhat.com/show_bug.cgi?id=1706759#c13

Comment 7 Lei Yang 2019-09-06 04:50:09 UTC
Also hit it with qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64,and no "return" event in qmp when unplug netdev device.
Version:
host:
qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64
kernel-4.18.0-141.el8.x86_64
guest:
win2019-x86_64

Comment 8 Lei Yang 2019-10-28 07:27:53 UTC
Hit this issue with RHEL7.7 guest.NO "timestamp" event in qmp when hot plug rtl8139/e1000 device and no "DEVICE_DELETED" event in qmp after unplug rtl8139/e1000 device.

host qemu-kvm-rhev version: qemu-kvm-1.5.3-167.el7_7.2.x86_64
host kernel version: kernel-3.10.0-1062.4.1.el7.x86_64
guest: RHEL7.7 guest

How reproducible:
3/3

Steps to Reproduce:
1.Boot guest up with vertio-net-pci device
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-machine pc  \
-nodefaults \
-device qxl-vga,bus=pci.0,addr=0x2 \
-device i6300esb,bus=pci.0,addr=0x3 \
-watchdog-action reset \
-device pci-bridge,id=pci_bridge,bus=pci.0,addr=0x4,chassis_nr=1 \
-device intel-hda,bus=pci.0,addr=0x5 \
-device hda-duplex  \
-device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
-device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
-device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
-device nec-usb-xhci,id=usb2,bus=pci.0,addr=0x7 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x8 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \
-device virtio-net-pci,mac=9a:74:72:62:2c:d7,id=idEA7CKP,netdev=idap2QoN,bus=pci.0,addr=0x9  \
-netdev tap,id=idap2QoN,vhost=on \
-m 11264  \
-smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
-cpu 'Opteron_G3',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb2.0,port=1  \
-spice port=3000,password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,strict=off,order=cdn,once=c  \
-no-hpet \
-vnc :0 \
-enable-kvm \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa \
-qmp tcp:0:5555,server,nowait \
-monitor stdio \

2. From qmp monitor, set_link false the virtio-net-pci device.
{'execute': 'set_link', 'arguments': {'name': 'idEA7CKP', 'up': false }}

3. Hot plug an rtl8139 device.(NO "timestamp" event in qmp when hot plug rtl8139/e1000 device)
{'execute': 'netdev_add', 'arguments': {'type': 'tap', 'id': 'id585VcC'}}
{"return": {}}
{'execute': 'device_add', 'arguments': {'addr': '0x6', 'netdev': 'id585VcC', 'bus': 'pci.0', 'driver': 'virtio-net-pci','mac': '9a:d9:f7:4a:51:ad', 'id': 'idScLb6u'}}
{"return": {}}

4. Hot unplug rtl8139 device.
{"execute":"device_del","arguments":{"id":"idScLb6u"}}
{"return": {}}
{"timestamp": {"seconds": 1572244736, "microseconds": 413379}, "event": "DEVICE_DELETED", "data": {"device": "idScLb6u", "path": "/machine/peripheral/idScLb6u"}}

5.Check the "event": "DEVICE_DELETED" logs, only return "/machine/peripheral/idScLb6u" delete info.It should return follow as well:
{"timestamp": {"seconds": 1572244691, "microseconds": 618294}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/idScLb6u/virtio-backend"}}
{"timestamp": {"seconds": 1572244691, "microseconds": 635212}, "event": "DEVICE_DELETED", "data": {"device": "idScLb6u", "path": "/machine/peripheral/idScLb6u"}}

6. Additional info:
1. It can not be reproduced with virtio-net-pci device.
2. e1000 device hit this issue.

Please tell me know,if we need open a new BZ on the rhel7 version.thanks a lot.

Comment 9 Yiqian Wei 2019-11-29 03:32:12 UTC
I can reproduce this bug on RHEL7.8 host 

host version:
qemu-kvm-rhev-2.12.0-38.el7.x86_64
kernel-3.10.0-1113.el7.x86_64
seabios-1.11.0-2.el7.x86_64
virtio-win-1.9.8-6.el7.noarch

guest:Win2019(q35 + seabios)

Comment 10 ybendito 2019-12-26 04:54:42 UTC
Posted upstream patch for virtio-net-pci only
https://lists.nongnu.org/archive/html/qemu-devel/2019-12/msg05198.html
I think similar issues with vfio/e1000e should be taken to separate BZ
Note also similar problem for w1000e in
https://bugzilla.redhat.com/show_bug.cgi?id=1738821

Comment 11 ybendito 2020-01-08 09:39:04 UTC
Applied in upstream

Comment 15 Ademar Reis 2020-02-05 22:57:42 UTC
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Comment 16 John Ferlan 2020-02-06 20:46:27 UTC
NB: Setting back to jusual as it looks like Amnon only meant to change the subcomponent, but didn't notice that doing that causes defaults to reset.

Comment 19 Lei Yang 2020-02-18 03:11:17 UTC
==Steps:

1.Boot win2019 guest.
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-machine q35 \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x2 \
-m 30720  \
-smp 16,maxcpus=16,cores=8,threads=1,sockets=2  \
-cpu 'EPYC',hv_stimer,hv_synic,hv_vpindex,hv_reset,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,+kvm_pv_unhalt \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,mac=9a:bf:e2:34:1b:a3,id=net1,netdev=hostnet1,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=hostnet1,vhost=on \
-drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/kvm_autotest_root/iso/windows/winutils.iso \
-device scsi-cd,id=cd1,drive=drive_cd1 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
-vnc :0  \
-rtc base=localtime,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \

==Verified with qemu-kvm-2.12.0-98.module+el8.2.0+5698+10a84757
2. From qmp monitor, unplug the virtio-net-pci device.
# telnet 10.73.224.24 5555
Trying 10.73.224.24...
Connected to 10.73.224.24.
Escape character is '^]'.
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 12, "major": 2}, "package": "qemu-kvm-2.12.0-98.module+el8.2.0+5698+10a84757"}, "capabilities": []}}
{"execute":"qmp_capabilities"}
{"return": {}}
{ 'execute': 'device_del', 'arguments': {'id': 'net1' }}
{"return": {}}
{"timestamp": {"seconds": 1581994458, "microseconds": 360232}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/net1/virtio-backend"}}
{"timestamp": {"seconds": 1581994458, "microseconds": 361651}, "event": "DEVICE_DELETED", "data": {"device": "net1", "path": "/machine/peripheral/net1"}}

3.Check the "event": "DEVICE_DELETED" logs, there are two delete info.

So this bug has been fixed very well in qemu-kvm-2.12.0-98.module+el8.2.0+5698+10a84757.

Comment 21 errata-xmlrpc 2020-04-28 15:32:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1587