Bug 1856678

Summary: [q35] No "DEVICE_DELETED" event in qmp after "device_del" virtio-blk-pci/virtio-scsi-pci
Product: Red Hat Enterprise Linux 9 Reporter: qing.wang <qinwang>
Component: qemu-kvmAssignee: Julia Suvorova <jusual>
qemu-kvm sub component: PCI QA Contact: qing.wang <qinwang>
Status: CLOSED WONTFIX Docs Contact:
Severity: high    
Priority: medium CC: ailan, coli, jinzhao, juzhang, lijin, menli, phou, qzhang, virt-maint
Version: unspecifiedKeywords: Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Windows   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-01-14 07:27:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1744438    

Description qing.wang 2020-07-14 08:20:57 UTC
Description of problem:
Hotplug multi disks into window guest then unplug them, there are no  DEVICE_DELETED" event for some disks in qmp after "device_del" virtio-blk-pci

Version-Release number of selected component (if applicable):
4.18.0-193.10.1.el8_2.x86_64
qemu-kvm-common-debuginfo-4.2.0-27.module+el8.2.1+7092+9d345e72.x86_64
qemu-kvm-core-4.2.0-27.module+el8.2.1+7092+9d345e72.x86_64


How reproducible:
80%

Steps to Reproduce:
1.create 20 images on host
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg0.qcow2 128M
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg1.qcow2 128M
...
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg19.qcow2 128M

2.boot vm 
 /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 14336  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
   \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/win2019-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:4e:43:cd:68:ce,id=idvHHXiI,netdev=idooVYVs,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idooVYVs,vhost=on \
    -blockdev node-name=file_cd1,driver=file,read-only=on,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \
    -device ide-cd,id=cd1,drive=drive_cd1,bootindex=1,write-cache=on,bus=ide.0,unit=0  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
    -device pcie-root-port,id=pcie_extra_root_port_1,addr=0x3.0x1,bus=pcie.0,chassis=6 \
    -device pcie-root-port,id=pcie_extra_root_port_2,addr=0x3.0x2,bus=pcie.0,chassis=7 \
    -device pcie-root-port,id=pcie_extra_root_port_3,addr=0x3.0x3,bus=pcie.0,chassis=8 \
    -device pcie-root-port,id=pcie_extra_root_port_4,addr=0x3.0x4,bus=pcie.0,chassis=9 \
    -device pcie-root-port,id=pcie_extra_root_port_5,addr=0x3.0x5,bus=pcie.0,chassis=10 \
    -device pcie-root-port,id=pcie_extra_root_port_6,addr=0x3.0x6,bus=pcie.0,chassis=11 \
    -device pcie-root-port,id=pcie_extra_root_port_7,addr=0x3.0x7,bus=pcie.0,chassis=12 \
    -device pcie-root-port,id=pcie_extra_root_port_8,multifunction=on,bus=pcie.0,addr=0x4,chassis=13 \
    -device pcie-root-port,id=pcie_extra_root_port_9,addr=0x4.0x1,bus=pcie.0,chassis=14 \
    -device pcie-root-port,id=pcie_extra_root_port_10,addr=0x4.0x2,bus=pcie.0,chassis=15 \
    -device pcie-root-port,id=pcie_extra_root_port_11,addr=0x4.0x3,bus=pcie.0,chassis=16 \
    -device pcie-root-port,id=pcie_extra_root_port_12,addr=0x4.0x4,bus=pcie.0,chassis=17 \
    -device pcie-root-port,id=pcie_extra_root_port_13,addr=0x4.0x5,bus=pcie.0,chassis=18 \
    -device pcie-root-port,id=pcie_extra_root_port_14,addr=0x4.0x6,bus=pcie.0,chassis=19 \
    -device pcie-root-port,id=pcie_extra_root_port_15,addr=0x4.0x7,bus=pcie.0,chassis=20 \
    -device pcie-root-port,id=pcie_extra_root_port_16,multifunction=on,bus=pcie.0,addr=0x5,chassis=21 \
    -device pcie-root-port,id=pcie_extra_root_port_17,addr=0x5.0x1,bus=pcie.0,chassis=22 \
    -device pcie-root-port,id=pcie_extra_root_port_18,addr=0x5.0x2,bus=pcie.0,chassis=23 \
    -device pcie-root-port,id=pcie_extra_root_port_19,addr=0x5.0x3,bus=pcie.0,chassis=24

3.hotplug disks
{"execute": "blockdev-add", "arguments": {"node-name": "file_stg0", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg0.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "GyzymquG"}
{"execute": "blockdev-add", "arguments": {"node-name": "drive_stg0", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg0"}, "id": "MTLAmOjL"}
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg0", "drive": "drive_stg0", "write-cache": "on", "bus": "pcie_extra_root_port_0", "addr": "0x0", "iothread": "iothread1"}, "id": "gyBtjBSf"}
...
{"execute": "blockdev-add", "arguments": {"node-name": "file_stg19", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg19.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "WLPRrShG"}
{"execute": "blockdev-add", "arguments": {"node-name": "drive_stg19", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg19"}, "id": "1ndpYYc5"}
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg19", "drive": "drive_stg19", "write-cache": "on", "bus": "pcie_extra_root_port_19", "addr": "0x0", "iothread": "iothread1"}, "id": "Ym6aR3NI"}

4.online above disks in guest

5.do some io on those disks in guest
D:\Iozone\iozone.exe -azR -r 64k -n 100M -g 100M -M -i 0 -i 1 -I -b iozone_F.xls -f F:\testfile

6.hotunplug above disks
{"execute": "device_del", "arguments": {"id": "stg0"}}
{"execute": "device_del", "arguments": {"id": "stg1"}}
..
{"execute": "device_del", "arguments": {"id": "stg19"}}


Actual results:
Some unpluged disks did no get DEVICE_DELETED event in qmp

Expected results:
each unpluged disk should get events like as:
 {"timestamp": {"seconds": 1594385560, "microseconds": 918101}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg0/virtio-backend"}}
 {"timestamp": {"seconds": 1594385560, "microseconds": 977383}, "event": "DEVICE_DELETED", "data": {"device": "stg0", "path": "/machine/peripheral/stg0"}}

Additional info:
This issue may reproduce via:
python ConfigTest.py --testcase=multi_disk_random_hotplug.serial.single_type.q35 --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=Win2019 --driveformat=virtio_blk --nicmodel=virtio_net --imageformat=qcow2 --machines=q35 --customsparams="qemu_force_use_drive_expression = no\nimage_aio=threads\ncd_format=ide" --clone=no

This issue also found on 
4.18.0-214.el8.x86_64
qemu-kvm-core-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64

Comment 1 Ademar Reis 2020-07-14 13:30:19 UTC
Is this a variation of Bug 1850412, which is about to be closed as invalid?

Comment 2 qing.wang 2020-07-15 05:31:20 UTC
(In reply to Ademar Reis from comment #1)
> Is this a variation of Bug 1850412, which is about to be closed as invalid?

Bug 1850412 related to hotunplug multifunction disk. this bug is similar as Bug 1810858.
It is old issue. 1810858 is for single disk hotunplug and can not reproduce on recent version.

Comment 3 qing.wang 2020-07-27 07:49:30 UTC
Hit same issue on 
4.18.0-226.el8.x86_64
qemu-kvm-core-4.2.0-30.module+el8.3.0+7298+c26a06b8.x86_64
seabios-1.13.0-1.module+el8.3.0+6423+e4cb6418.x86_64

It may reproduced via:

python ConfigTest.py --testcase=block_hotplug.block_virtio.fmt_qcow2.default.with_plug.default.multi_pci.q35 --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=Win2019 --driveformat=virtio_blk --nicmodel=virtio_net --imageformat=qcow2 --machines=q35 --customsparams="image_aio=threads\ncd_format=ide"

Comment 6 qing.wang 2021-02-07 02:57:20 UTC
Hit same issue on win2019 
{'kvm_version': '4.18.0-268.el8.x86_64', 'qemu_version': 'qemu-kvm-core-5.2.0-2.module+el8.4.0+9186+ec44380f.x86_64'}

reproduced on automation:
python ConfigTest.py --testcase=block_multifunction.default.q35  --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=Win2019 --driveformat=virtio_blk --nicmodel=virtio_net --imageformat=qcow2 --machines=q35  --firmware=ovmf --customsparams="vm_mem_limit = 12G\nimage_aio=native\ncd_format=ide"

http://fileshare.englab.nay.redhat.com/pub/section2/images_backup/qlogs/%5bQinwang%5d%5bW53%5d%5bVirtio_blk%5d%5b2019%5d-%5b8.4-AV%5d-11-Q35+Ovmf+Win2019+Qcow2+Virtio_blk+Local+aio_native+qemu-5.2/test-results/158-Host_RHEL.m8.u4.product_av.ovmf.qcow2.virtio_blk.up.virtio_net.Guest.Win2019.x86_64.io-github-autotest-qemu.block_multifunction.q35/

Comment 8 John Ferlan 2021-09-09 13:56:34 UTC
Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.

Comment 9 Peixiu Hou 2021-11-15 09:09:25 UTC
Hit the same issue with virtio-scsi-pci on RHEL9 host.
Guest os: Win8.1-64
kernel-5.14.0-10.el9.x86_64
qemu-kvm-6.1.0-5.el9.x86_64
virtio-win-prewhql-0.1-214
seabios-bin-1.14.0-7.el9.noarch

reproduced on automation:
python ConfigTest.py --testcase=block_hotplug.block_scsi.fmt_qcow2.with_plug.with_reboot.one_pci --platform=x86_64 --guestname=Win8..1 --driveformat=virtio_scsi --machines=q35

Comment 11 RHEL Program Management 2022-01-14 07:27:03 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.