Bug 1790899 - [RFE] QEMU devices should have the option to enable/disable hotplug/unplug
Summary: [RFE] QEMU devices should have the option to enable/disable hotplug/unplug
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.1
Hardware: x86_64
OS: Windows
high
high
Target Milestone: rc
: 8.0
Assignee: Julia Suvorova
QA Contact: jingzhao
URL:
Whiteboard:
Depends On:
Blocks: 1802592 1823805
TreeView+ depends on / blocked
 
Reported: 2020-01-14 13:57 UTC by David Critch
Modified: 2020-12-20 06:42 UTC (History)
30 users (show)

Fixed In Version: qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1802592 (view as bug list)
Environment:
Last Closed: 2020-07-28 07:12:15 UTC
Type: Feature Request
Target Upstream Version:


Attachments (Terms of Use)
after eject (162.69 KB, image/png)
2020-05-07 09:56 UTC, qing.wang
no flags Details
eject list (179.39 KB, image/png)
2020-05-07 09:58 UTC, qing.wang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:3172 0 None None None 2020-07-28 07:13:29 UTC

Description David Critch 2020-01-14 13:57:56 UTC
Description of problem:
We have a customer that is using Windows guests with libvirt (CNV) with virtio drivers loaded on the guest. Non-admin users are seeing the options to 'safely remove ...' the NIC, disks, etc. They'd like this functionality disabled.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Install a Windows 10 guest with virtio drivers
2. Login as a non-admin user
3. Select 'Eject SAS Controller' from taskbar


Actual results:
Controller is removed

Expected results:
There should be an option to prevent non-admin users from such tasks.

Additional info:
1. Microsoft indicate that this has to be done in the driver, as opposed to registry changes to prevent it
2. In VMWare there is an option, devices.hotplug = “false”, that currently works for the customer

Comment 5 Vadim Rozenfeld 2020-01-14 22:42:58 UTC
IIUC, group policy should be applied to manage such kind of things.
We can also check if SDDL  ( https://docs.microsoft.com/en-us/windows/win32/setupapi/specifying-a-security-descriptor-from-an-inf-file )
can be helpful in this case.

Comment 30 Daniel Berrangé 2020-02-04 18:54:40 UTC
Upstream public thread discussing options for QEMU/libvirt config here https://www.redhat.com/archives/libvir-list/2020-February/msg00110.html

Comment 45 Julia Suvorova 2020-02-19 15:07:37 UTC
Posted upstream (qemu-devel):
 [PATCH v2] pcie_root_port: Add enable_hotplug option

Comment 48 Julia Suvorova 2020-03-10 14:44:52 UTC
Merged upstream (commit 530a0963184e57e71a5b538e9161f115df533e96). Will be part of qemu-5.0.

Comment 52 Peixiu Hou 2020-03-31 07:45:49 UTC
Hi,

I tested on virtio-scsi-pci with hotplug=on/off option, tested with latest virtio-win driver(build 180 for vioscsi):

1. Tested with hotplug=on, command line as follows, in guest os "Safly Remove Hardware and eject Media" option,  there isn't "Eject Red Hat Virtio SCSI pass-through controller", only have "Eject Linux scsi_debug SCSI Disk Device" option, tried choose "Eject Linux scsi_debug SCSI Disk Device" option, the controller "Red Hat Virtio SCSI pass-through controller" will disappeared in device managerment.
=====================================================================================================
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie_extra_root_port_0,addr=0x0,hotplug=on \
    -blockdev node-name=file_stg0,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg0,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg0 \
    -device scsi-hd,id=stg01,drive=drive_stg0,write-cache=on,bus=virtio_scsi_pci1.0 \
=====================================================================================================

2. Tested with hotplug=off, command line as follows, the "Eject Linux scsi_debug SCSI Disk Device" option also can be executed, the controller "Red Hat Virtio SCSI pass-through controller" disappeared in device management as well after that.
=====================================================================================================
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
    -device virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie_extra_root_port_0,addr=0x0,hotplug=off \
    -blockdev node-name=file_stg0,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg0,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg0 \
    -device scsi-hd,id=stg01,drive=drive_stg0,write-cache=on,bus=virtio_scsi_pci1.0 \
=====================================================================================================

3. And for virtio-scsi-pci controller that installed guest os system(only add 1 scsi controller for system disk, no data disk attanched), no matter hotplug=on or off, it will not allow to eject in guest os, in "Safly Remove Hardware and eject Media" option, C and S partition shown gray, and none "Eject Linux scsi_debug SCSI Disk Device" for this situation.

Used versions:
kernel-4.18.0-190.el8.x86_64
qemu-kvm-5.0.0-0.scrmod+el8.2.0+6108+d0330baa.rc0.x86_64
virtio-win-1.9.11-1.el8.iso
seabios-1.13.0-1.scrmod+el8.2.0+6108+d0330baa.x86_64

Best Regards~
Peixiu

Comment 54 Julia Suvorova 2020-03-31 13:05:52 UTC
(In reply to Peixiu Hou from comment #52)
> Hi,
> 
> I tested on virtio-scsi-pci with hotplug=on/off option, tested with latest
> virtio-win driver(build 180 for vioscsi):
> 
> 1. Tested with hotplug=on, command line as follows, in guest os "Safly
> Remove Hardware and eject Media" option,  there isn't "Eject Red Hat Virtio
> SCSI pass-through controller", only have "Eject Linux scsi_debug SCSI Disk
> Device" option, tried choose "Eject Linux scsi_debug SCSI Disk Device"
> option, the controller "Red Hat Virtio SCSI pass-through controller" will
> disappeared in device managerment.
> =============================================================================
> ========================
>     -device
> pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,
> addr=0x3,chassis=5 \
>     -device
> virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie_extra_root_port_0,addr=0x0,
> hotplug=on \


You need to disable hotplug on pcie-root-port instead of device itself, like this:
    -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x3,chassis=5,hotplug=off

This way you will not see SCSI Disk Device listed for eject.

Best regards, Julia Suvorova.


>     -blockdev
> node-name=file_stg0,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg0,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg0 \
>     -device
> scsi-hd,id=stg01,drive=drive_stg0,write-cache=on,bus=virtio_scsi_pci1.0 \
> =============================================================================
> ========================
> 
> 2. Tested with hotplug=off, command line as follows, the "Eject Linux
> scsi_debug SCSI Disk Device" option also can be executed, the controller
> "Red Hat Virtio SCSI pass-through controller" disappeared in device
> management as well after that.
> =============================================================================
> ========================
>     -device
> pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,
> addr=0x3,chassis=5 \
>     -device
> virtio-scsi-pci,id=virtio_scsi_pci1,bus=pcie_extra_root_port_0,addr=0x0,
> hotplug=off \
>     -blockdev
> node-name=file_stg0,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg0,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg0 \
>     -device
> scsi-hd,id=stg01,drive=drive_stg0,write-cache=on,bus=virtio_scsi_pci1.0 \
> =============================================================================
> ========================
> 
> 3. And for virtio-scsi-pci controller that installed guest os system(only
> add 1 scsi controller for system disk, no data disk attanched), no matter
> hotplug=on or off, it will not allow to eject in guest os, in "Safly Remove
> Hardware and eject Media" option, C and S partition shown gray, and none
> "Eject Linux scsi_debug SCSI Disk Device" for this situation.
> 
> Used versions:
> kernel-4.18.0-190.el8.x86_64
> qemu-kvm-5.0.0-0.scrmod+el8.2.0+6108+d0330baa.rc0.x86_64
> virtio-win-1.9.11-1.el8.iso
> seabios-1.13.0-1.scrmod+el8.2.0+6108+d0330baa.x86_64
> 
> Best Regards~
> Peixiu

Comment 56 Peixiu Hou 2020-04-01 03:12:13 UTC
Thanks Julia Suvorova~

I tested with command -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x3,chassis=5,hotplug=off, yes, none SCSI Disk Device listed for eject.
And changed with command -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x3,chassis=5,hotplug=on, the SCSI Disk Device is listed for eject, and can be ejected success.

Best Regards~
Peixiu

Comment 57 xiagao 2020-04-01 06:30:25 UTC
Hi Julia,

Test with balloon device with the following cmd.
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x3.0x5,hotplug=off \
-device virtio-balloon-pci,id=balloon0,bus=pci.5 \

And try to unplug it via qmp, if there can be some warning/info after we send unplug command?
{"execute": "device_del", "arguments": {"id": "balloon0"}}
{"return": {}}
{"execute": "device_del", "arguments": {"id": "balloon0"}}
{"error": {"class": "GenericError", "desc": "Device balloon0 is already in the process of unplug"}}

After that, the device is still in guest.

BR,
Xiaoling Gao

Comment 58 jingzhao 2020-04-01 07:37:09 UTC
Hi Julia

Based on comment 54 ~ 56

1. The usage of hotplug is: -device pcie-root-port, .... hotplug=on/off. we don't need to care about the device parameter of hotplug, am I right?

2. Did this support both windows and linux driver? 

3. This issue is for qemu bz fix, do we need to open bz for driver fix? 

4. QE now are preparing for test cases and gating automation test loop, could you provide QE a scratch build of 8.2.1 for pre-testing to get the correct expected result? 


Thanks very much

Comment 60 Julia Suvorova 2020-04-02 15:30:48 UTC
(In reply to xiagao from comment #57)
> Hi Julia,
> 
> Test with balloon device with the following cmd.
> -device
> pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x3.0x5,
> hotplug=off \
> -device virtio-balloon-pci,id=balloon0,bus=pci.5 \
> 
> And try to unplug it via qmp, if there can be some warning/info after we
> send unplug command?
> {"execute": "device_del", "arguments": {"id": "balloon0"}}
> {"return": {}}
> {"execute": "device_del", "arguments": {"id": "balloon0"}}
> {"error": {"class": "GenericError", "desc": "Device balloon0 is already in
> the process of unplug"}}
> 
> After that, the device is still in guest.
> 
> BR,
> Xiaoling Gao

Currently there's no warning. I'll add one. This way we can also avoid the
error message you get (the device still behaves properly, but there's an
extra flag that we turned on when was deleting the first time, and didn't
turn off because we're still not going to delete it).

Best regards, Julia Suvorova.

Comment 63 qing.wang 2020-04-03 06:49:22 UTC
Test for virtio-blk
version:
4.18.0-193.el8.x86_64
qemu-kvm-core-5.0.0-0.scrmod+el8.2.0+6151+7a36dcb6.wrb200401.x86_64



1.create images with different size
 qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg1.qcow2 1G
 qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg2.qcow2 2G
 ...
 qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg8.qcow2 8G
 
 2.boot vm
 /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -object iothread,id=iothread0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci,iothread=iothread0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:ab:fa:09:62:1c,id=idLVRSKI,netdev=idfMN0PL,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idfMN0PL,vhost=on \
    -m 8G  \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :6  \
    -qmp tcp:0:5956,server,nowait \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -blockdev node-name=file_stg1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg1.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg1 \
    -blockdev node-name=file_stg2,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg2.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg2,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg2 \
    \
    -blockdev node-name=file_stg3,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg3.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg3,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg3 \
    -blockdev node-name=file_stg4,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg4.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg4,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg4 \
    \
    -blockdev node-name=file_stg5,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg5.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg5,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg5 \
    -blockdev node-name=file_stg6,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg6.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg6,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg6 \
    \
    -blockdev node-name=file_stg7,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg7.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg7,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg7 \
    -blockdev node-name=file_stg8,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg8.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg8,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg8 \
    \
    -device pcie-root-port,id=pcie_port_1,slot=5,chassis=5,addr=0x5,bus=pcie.0,hotplug=off \
    -device virtio-blk-pci,id=stg1,drive=drive_stg1,bus=pcie_port_1,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_2,slot=6,chassis=6,addr=0x6,bus=pcie.0,hotplug=on \
    -device virtio-blk-pci,id=stg2,drive=drive_stg2,bus=pcie_port_2,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_3,slot=7,chassis=7,addr=0x7,bus=pcie.0,hotplug=off \
    -device pcie-root-port,id=pcie_port_4,slot=8,chassis=8,addr=0x8,bus=pcie.0,hotplug=on \



3 check disks and pci in guest
[root@bootp-73-131-155 ~]# lsblk
...
vda                               252:0    0   1G  0 disk 
vdb                               252:16   0   2G  0 disk 
[root@bootp-73-131-155 ~]# lspci
...
04:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
05:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)

4.hotplug disks

{"execute":"qmp_capabilities"}
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg3", "drive": "drive_stg3", "bus": "pcie_port_3"}, "id": "t07OBwF3"}
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg4", "drive": "drive_stg4", "bus": "pcie_port_4"}, "id": "t07OBwF4"}

5.check disks in guest
lsblk:
vda                               252:0    0   1G  0 disk 
vdb                               252:16   0   2G  0 disk 
vdc                               252:32   0   4G  0 disk   (new add)
lspci:
04:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
05:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
07:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01) (new add)
 
 info block in hmp
drive_stg1: /home/kvm_autotest_root/images/stg1.qcow2 (qcow2)
    Attached to:      /machine/peripheral/stg1/virtio-backend
    Cache mode:       writeback, direct

drive_stg2: /home/kvm_autotest_root/images/stg2.qcow2 (qcow2)
    Attached to:      /machine/peripheral/stg2/virtio-backend
    Cache mode:       writeback, direct

drive_stg3: /home/kvm_autotest_root/images/stg3.qcow2 (qcow2)
    Attached to:      /machine/peripheral/stg3/virtio-backend
    Cache mode:       writeback, direct

drive_stg4: /home/kvm_autotest_root/images/stg4.qcow2 (qcow2)
    Attached to:      /machine/peripheral/stg4/virtio-backend
    Cache mode:       writeback, direct

6.delete all disks 
 
{"execute":"qmp_capabilities"}

{"execute": "device_del", "arguments": {"id": "stg1"}, "id": "XVosfhH1"}
{"execute": "device_del", "arguments": {"id": "stg2"}, "id": "XVosfhH2"}

{"execute": "device_del", "arguments": {"id": "stg3"}, "id": "XVosfhH3"}
{"execute": "device_del", "arguments": {"id": "stg4"}, "id": "XVosfhH4"}


7.check disk in guest
[root@bootp-73-131-155 ~]# lsblk
...
vda                               252:0    0   1G  0 disk 

8.check disk in hmp
info block
(qemu) info block
drive_image1 (#block162): /home/kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2 (qcow2)
    Attached to:      image1
    Cache mode:       writeback, direct

drive_stg1: /home/kvm_autotest_root/images/stg1.qcow2 (qcow2)
    Attached to:      /machine/peripheral/stg1/virtio-backend
    Cache mode:       writeback, direct

drive_stg3: /home/kvm_autotest_root/images/stg3.qcow2 (qcow2)
    Attached to:      /machine/peripheral/stg3/virtio-backend
    Cache mode:       writeback, direct
(qemu) 


Conclusion: 
it takes effect when hotplug=off on its controller
info block in hmp is incorrect. it is not sync with guest.

Comment 64 CongLi 2020-04-03 08:51:24 UTC
Hi Julia, 

QE met some issues during testing, could you please help confirm ? Thanks.

1. hotplug=off and do unplug operation 2 times.
   The 1st unplug operation is expected, but when tried again for 2nd time, there is error 'Device rng0 is already in the process of unplug', which causes confusion if the 1st unplug has been done.

{"execute":"device_del","arguments":{"id":"rng0"}}
{"return": {}}   --> unplug does not work
{"execute":"device_del","arguments":{"id":"rng0"}}
{"error": {"class": "GenericError", "desc": "Device rng0 is already in the process of unplug"}}

2. hotplug=off works for virtio-scsi-pci controller, but does not work for the device under it.

Recording test result from qinwang@.
"""
1.create images with different size
 qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg1.qcow2 1G
 qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg2.qcow2 2G
 ...
 qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg8.qcow2 8G
 
 2.boot vm
 /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -object iothread,id=iothread0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci,iothread=iothread0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:ab:fa:09:62:1c,id=idLVRSKI,netdev=idfMN0PL,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idfMN0PL,vhost=on \
    -m 8G  \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :6  \
    -qmp tcp:0:5956,server,nowait \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -blockdev node-name=file_stg1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg1.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg1 \
    -blockdev node-name=file_stg2,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg2.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg2,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg2 \
    \
    -blockdev node-name=file_stg3,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg3.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg3,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg3 \
    -blockdev node-name=file_stg4,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg4.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg4,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg4 \
    \
    -blockdev node-name=file_stg5,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg5.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg5,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg5 \
    -blockdev node-name=file_stg6,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg6.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg6,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg6 \
    \
    -blockdev node-name=file_stg7,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg7.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg7,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg7 \
    -blockdev node-name=file_stg8,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg8.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg8,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg8 \
    \
    -device pcie-root-port,id=pcie_port_1,slot=5,chassis=5,addr=0x5,bus=pcie.0,hotplug=off \
    -device virtio-scsi-pci,id=scsi1,bus=pcie_port_1,addr=0x0,hotplug=off \
    -device scsi-hd,id=stg1,drive=drive_stg1,write-cache=on,bus=scsi1.0 \
    \
    -device pcie-root-port,id=pcie_port_2,slot=6,chassis=6,addr=0x6,bus=pcie.0,hotplug=off \
    -device virtio-scsi-pci,id=scsi2,bus=pcie_port_2,addr=0x0,hotplug=on \
    -device scsi-hd,id=stg2,drive=drive_stg2,write-cache=on,bus=scsi2.0 \
    \
    -device pcie-root-port,id=pcie_port_3,slot=7,chassis=7,addr=0x7,bus=pcie.0,hotplug=on \
    -device virtio-scsi-pci,id=scsi3,bus=pcie_port_3,addr=0x0,hotplug=off \
    -device scsi-hd,id=stg3,drive=drive_stg3,write-cache=on,bus=scsi3.0 \
    \
    -device pcie-root-port,id=pcie_port_4,slot=8,chassis=8,addr=0x8,bus=pcie.0,hotplug=on \
    -device virtio-scsi-pci,id=scsi4,bus=pcie_port_4,addr=0x0,hotplug=on \
    -device scsi-hd,id=stg4,drive=drive_stg4,write-cache=on,bus=scsi4.0 \
    \
    -device pcie-root-port,id=pcie_port_5,slot=9,chassis=9,addr=0x9,bus=pcie.0,hotplug=off \
    -device virtio-scsi-pci,id=scsi5,bus=pcie_port_5,addr=0x0 \
    -device scsi-hd,id=stg5,drive=drive_stg5,write-cache=on,bus=scsi5.0 \
    \
    -device pcie-root-port,id=pcie_port_6,slot=10,chassis=10,addr=0x10,bus=pcie.0,hotplug=on \
    -device virtio-scsi-pci,id=scsi6,bus=pcie_port_6,addr=0x0 \
    -device scsi-hd,id=stg6,drive=drive_stg6,write-cache=on,bus=scsi6.0 \
    \
    -device pcie-root-port,id=pcie_port_7,slot=11,chassis=11,addr=0x11,bus=pcie.0 \
    -device virtio-scsi-pci,id=scsi7,bus=pcie_port_7,addr=0x0,hotplug=off \
    -device scsi-hd,id=stg7,drive=drive_stg7,write-cache=on,bus=scsi7.0 \
    \
    -device pcie-root-port,id=pcie_port_8,slot=12,chassis=12,addr=0x12,bus=pcie.0 \
    -device virtio-scsi-pci,id=scsi8,bus=pcie_port_8,addr=0x0,hotplug=on \
    -device scsi-hd,id=stg8,drive=drive_stg8,write-cache=on,bus=scsi8.0 \


3 check disk in guest
[root@bootp-73-131-155 ~]# lsblk
...
sdb                                 8:16   0   1G  0 disk
sdc                                 8:32   0   2G  0 disk
sdd                                 8:48   0   3G  0 disk
sde                                 8:64   0   4G  0 disk
sdf                                 8:80   0   5G  0 disk
sdg                                 8:96   0   6G  0 disk
sdh                                 8:112  0   7G  0 disk
sdi                                 8:128  0   8G  0 disk


4.delete all disks
 
{"execute":"qmp_capabilities"}

{"execute": "device_del", "arguments": {"id": "stg1"}, "id": "XVosfhH1"}
{"execute": "device_del", "arguments": {"id": "stg2"}, "id": "XVosfhH2"}

{"execute": "device_del", "arguments": {"id": "stg3"}, "id": "XVosfhH3"}
{"execute": "device_del", "arguments": {"id": "stg4"}, "id": "XVosfhH4"}

{"execute": "device_del", "arguments": {"id": "stg5"}, "id": "XVosfhH5"}
{"execute": "device_del", "arguments": {"id": "stg6"}, "id": "XVosfhH6"}

{"execute": "device_del", "arguments": {"id": "stg7"}, "id": "XVosfhH7"}
{"execute": "device_del", "arguments": {"id": "stg8"}, "id": "XVosfhH8"}

5.check disk in guest
[root@bootp-73-131-155 ~]# lsblk
...
sdb                                 8:16   0   1G  0 disk
sdd                                 8:48   0   3G  0 disk
sdh                                 8:112  0   7G  0 disk


Conclusion: it takes effect when hotplug=off on device virtio-scsi-pci instead of pcie-root-port
"""

Thanks.

Comment 68 Daniel Berrangé 2020-04-06 14:03:59 UTC
(In reply to Julia Suvorova from comment #66)
> This raises two questions:
> 1. Do we need to forbid unplug for every child bus if it's connected to pci?
> Like, pcie - pcie usb bridge - usb device
> (you can't unplug usb bridge, but you can unplug usb device now)
> This may (or may not) require support of hotplug disabling for all buses.

Conceptually if the bus (USB, SCSI) has support for disabling hotplug, it is desirable for QEMU to support configuring that. For the context of this bug though, I think we should focus exclusively on PCI, otherwise we'll be overloading multiple distinct RFEs into a single bug. IOW if there is a need to disable hotplug on other buses, we should have distinct bugs, one for each bus that needs it.

> 2. Do we need to restrict unplug not only from guest, but from qemu via qmp
> too?
> I don't see usecase for this, and it's not needed for this bug.

In general QEMU suffers greatly from situations like this where we don't do good error reporting/handling. It leads to a support burden as it is hard to diagnose problems when applications or users do unexpected things.

Thus I think it is important that if we've disabled hotplug on a slot, then an attempt to plug/unplug in that slot via QMP should give a clear error message that hotplug is disabled.

Comment 69 Julia Suvorova 2020-04-06 16:34:15 UTC
Getting to the conclusion,
1. Any child devices cannot be unplugged through guest, if it's disabled on root port.
This needs to be carefully checked, as we rely on guest behaviour.

2. Only devices connected to pci directly couldn't be (un)plugged via qmp, if it's disabled on root port.
device_add(del) raises an error (included in the build I mentioned, but not in upstream due to softfreeze).

Comment 83 Peixiu Hou 2020-05-07 08:56:54 UTC
Hi,

Tests on new qemu version: qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64

1. Tested this issue with virtio-scsi device on win10-64 guest, test steps same as comment#62, got the same result as well.

2. Tested this issue with virtio-input device on win10-64 guest, tested commands as follows:
=============================================================================================
    -device pcie-root-port,id=pcie_port_1,slot=5,chassis=5,addr=0x5,bus=pcie.0,hotplug=off \
    -device virtio-keyboard-pci,id=kbd0,bus=pcie_port_1,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_3,slot=7,chassis=7,addr=0x7,bus=pcie.0,hotplug=on \
    -device virtio-keyboard-pci,id=kbd1,bus=pcie_port_3,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_2,slot=6,chassis=6,addr=0x6,bus=pcie.0,hotplug=off \
    -device virtio-mouse-pci,id=mouse0,bus=pcie_port_2,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_4,slot=8,chassis=8,addr=0x8,bus=pcie.0,hotplug=on \
    -device virtio-mouse-pci,id=mouse1,bus=pcie_port_4,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_5,slot=9,chassis=9,addr=0x9,bus=pcie.0,hotplug=off \
    -device virtio-tablet-pci,id=tablet0,bus=pcie_port_5,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_6,slot=10,chassis=10,addr=0x10,bus=pcie.0,hotplug=on \
    -device virtio-tablet-pci,id=tablet1,bus=pcie_port_6,addr=0x0 \
==============================================================================================
With hotplug=off on pcie bus, device kbd0, mouse0, and tablet0 cannot be unplugged.
With hotplug=on on pcie bus, device kbd1, mouse1, and tablet1 can be eject smoothly on guest "Safely remove hardware and media" options.

Best Regards~
Peixiu

Comment 84 qing.wang 2020-05-07 09:56:54 UTC
Created attachment 1686109 [details]
after eject

Comment 85 qing.wang 2020-05-07 09:58:14 UTC
Created attachment 1686111 [details]
eject list

Comment 86 qing.wang 2020-05-07 10:14:25 UTC
Tests on new qemu version: qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64
Guest win2019

For virtio-scsi, there are some issues:

/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -object iothread,id=iothread0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci,iothread=iothread0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:ab:fa:09:62:1c,id=idLVRSKI,netdev=idfMN0PL,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idfMN0PL,vhost=on \
    -m 8G  \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :6  \
    -qmp tcp:0:5956,server,nowait \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -blockdev node-name=file_stg1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg1.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg1 \
    -blockdev node-name=file_stg2,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg2.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg2,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg2 \
    \
    -blockdev node-name=file_stg3,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg3.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg3,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg3 \
    -blockdev node-name=file_stg4,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg4.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg4,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg4 \
    \
    -blockdev node-name=file_stg5,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg5.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg5,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg5 \
    -blockdev node-name=file_stg6,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg6.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg6,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg6 \
    \
    -blockdev node-name=file_stg7,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg7.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg7,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg7 \
    -blockdev node-name=file_stg8,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg8.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg8,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg8 \
    \
    -device pcie-root-port,id=pcie_port_1,slot=5,chassis=5,addr=0x5,bus=pcie.0,hotplug=off \
    -device virtio-scsi-pci,id=scsi1,bus=pcie_port_1,addr=0x0,hotplug=off \
    -device scsi-hd,id=stg1,drive=drive_stg1,write-cache=on,bus=scsi1.0 \
    \
    -device pcie-root-port,id=pcie_port_2,slot=6,chassis=6,addr=0x6,bus=pcie.0,hotplug=off \
    -device virtio-scsi-pci,id=scsi2,bus=pcie_port_2,addr=0x0,hotplug=on \
    -device scsi-hd,id=stg2,drive=drive_stg2,write-cache=on,bus=scsi2.0 \
    \
    -device pcie-root-port,id=pcie_port_3,slot=7,chassis=7,addr=0x7,bus=pcie.0,hotplug=on \
    -device virtio-scsi-pci,id=scsi3,bus=pcie_port_3,addr=0x0,hotplug=off \
    -device scsi-hd,id=stg3,drive=drive_stg3,write-cache=on,bus=scsi3.0 \
    \
    -device pcie-root-port,id=pcie_port_4,slot=8,chassis=8,addr=0x8,bus=pcie.0,hotplug=on \
    -device virtio-scsi-pci,id=scsi4,bus=pcie_port_4,addr=0x0,hotplug=on \
    -device scsi-hd,id=stg4,drive=drive_stg4,write-cache=on,bus=scsi4.0 \
    \
    -device pcie-root-port,id=pcie_port_5,slot=9,chassis=9,addr=0x9,bus=pcie.0,hotplug=off \
    \
    -device pcie-root-port,id=pcie_port_6,slot=10,chassis=10,addr=0x10,bus=pcie.0,hotplug=off \
    \
    -device pcie-root-port,id=pcie_port_7,slot=11,chassis=11,addr=0x11,bus=pcie.0 \
    \
    -device pcie-root-port,id=pcie_port_8,slot=12,chassis=12,addr=0x12,bus=pcie.0 \

There are 4 disks D(1G) E(2G) F(3G) G(4G) in guest.

D E not in list,it is good.
FG are list, i think just G in the list.
And if we try to remove them (F or G), It is disappear in list, but it exist in explorer. And the application become non responding like as file explorer and disk management.


For virtio-blk, it looks good.

/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -object iothread,id=iothread0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci,iothread=iothread0,bus=pcie.0-root-port-3,addr=0x0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/win2019-64-virtio.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:ab:fa:09:62:1c,id=idLVRSKI,netdev=idfMN0PL,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idfMN0PL,vhost=on \
    -m 8G  \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :6  \
    -qmp tcp:0:5956,server,nowait \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -blockdev node-name=file_stg1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg1.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg1 \
    -blockdev node-name=file_stg2,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg2.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg2,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg2 \
    \
    -blockdev node-name=file_stg3,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg3.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg3,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg3 \
    -blockdev node-name=file_stg4,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg4.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg4,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg4 \
    \
    -blockdev node-name=file_stg5,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg5.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg5,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg5 \
    -blockdev node-name=file_stg6,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg6.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg6,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg6 \
    \
    -blockdev node-name=file_stg7,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg7.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg7,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg7 \
    -blockdev node-name=file_stg8,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/stg8.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg8,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg8 \
    \
    -device pcie-root-port,id=pcie_port_1,slot=5,chassis=5,addr=0x5,bus=pcie.0,hotplug=off \
    -device virtio-blk-pci,id=stg1,drive=drive_stg1,bus=pcie_port_1,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_2,slot=6,chassis=6,addr=0x6,bus=pcie.0,hotplug=on \
    -device virtio-blk-pci,id=stg2,drive=drive_stg2,bus=pcie_port_2,addr=0x0 \
    \
    -device pcie-root-port,id=pcie_port_3,slot=7,chassis=7,addr=0x7,bus=pcie.0,hotplug=off \
    -device pcie-root-port,id=pcie_port_4,slot=8,chassis=8,addr=0x8,bus=pcie.0,hotplug=on \

Comment 87 qing.wang 2020-05-08 01:32:20 UTC
Hi,Julia,Please help to check comment 86,if it is a bug, do we need to open a new bug track it?

Comment 88 Lei Yang 2020-05-12 04:12:37 UTC
Hi, Julia

I test params "hotplug=off" on qemu version qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64
Guest RHEL.8.2

For virtual network has some issue.
QEMU CLI:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1'  \
-sandbox on  \
-machine q35 \
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
-device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
-nodefaults \
-device VGA,bus=pcie.0,addr=0x2 \
-m 6144  \
-smp 6,maxcpus=6,cores=3,threads=1,dies=1,sockets=2  \
-cpu 'Haswell-noTSX',+kvm_pv_unhalt \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
-device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
-blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
-blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
-device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
-device pcie-root-port,id=pcie-root-port-3,hotplug=off,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,mac=9a:82:59:0b:01:17,id=idVn2ph4,mq=on,vectors=10,netdev=idb8GVnn,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idb8GVnn,vhost=on \
-vnc :0  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \

===>steps
1. Test inside the guest.
# lspci |grep Eth
05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
# echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/remove

2. The nic has been removed.
# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 60  bytes 5336 (5.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 60  bytes 5336 (5.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:fe:a5:ed  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This is not expected result.

Best regards
Lei Yang

Comment 89 Yumei Huang 2020-05-12 08:28:19 UTC
Hit same issue with virtio-balloon device as comment 88. The device comes back after reboot guest.

Comment 90 Julia Suvorova 2020-05-13 16:05:17 UTC
(In reply to Lei Yang from comment #88)
> Hi, Julia
> 
> I test params "hotplug=off" on qemu version
> qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64
> Guest RHEL.8.2
> 
> For virtual network has some issue.
> QEMU CLI:
> /usr/libexec/qemu-kvm \
> -name 'avocado-vt-vm1'  \
> -sandbox on  \
> -machine q35 \
> -device
> pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,
> chassis=1 \
> -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
> -nodefaults \
> -device VGA,bus=pcie.0,addr=0x2 \
> -m 6144  \
> -smp 6,maxcpus=6,cores=3,threads=1,dies=1,sockets=2  \
> -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
> -device
> pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,
> chassis=2 \
> -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
> -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
> -device
> pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,
> chassis=3 \
> -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
> -blockdev
> node-name=file_image1,driver=file,aio=threads,filename=/home/
> kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2,cache.direct=on,cache.
> no-flush=off \
> -blockdev
> node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_image1 \
> -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
> -device
> pcie-root-port,id=pcie-root-port-3,hotplug=off,port=0x3,addr=0x1.0x3,
> bus=pcie.0,chassis=4 \
> -device
> virtio-net-pci,mac=9a:82:59:0b:01:17,id=idVn2ph4,mq=on,vectors=10,
> netdev=idb8GVnn,bus=pcie-root-port-3,addr=0x0  \
> -netdev tap,id=idb8GVnn,vhost=on \
> -vnc :0  \
> -rtc base=utc,clock=host,driftfix=slew  \
> -boot menu=off,order=cdn,once=c,strict=off \
> -enable-kvm \
> -device
> pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,
> addr=0x3,chassis=5 \
> -monitor stdio \
> -qmp tcp:0:5555,server,nowait \
> 
> ===>steps
> 1. Test inside the guest.
> # lspci |grep Eth
> 05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
> # echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/remove
> 
> 2. The nic has been removed.
> # ifconfig
> lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
>         inet 127.0.0.1  netmask 255.0.0.0
>         inet6 ::1  prefixlen 128  scopeid 0x10<host>
>         loop  txqueuelen 1000  (Local Loopback)
>         RX packets 60  bytes 5336 (5.2 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 60  bytes 5336 (5.2 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
>         inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
>         ether 52:54:00:fe:a5:ed  txqueuelen 1000  (Ethernet)
>         RX packets 0  bytes 0 (0.0 B)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 0  bytes 0 (0.0 B)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> This is not expected result.

'echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/remove' nor the same command with 'hotplug=on' device will lead to removing device from QEMU (no DEVICE_DELETED event).
This command removes a device only inside Linux, and you can see it back if you rescan: 'echo 1 > /sys/bus/pci/rescan'.
This behaviour is correct.

If you want to actually remove a device, you need to power down the slot that your device inserted to:
'echo 0 > /sys/bus/pci/slots/`slot_number`/power'

Comment 91 Julia Suvorova 2020-05-13 16:14:42 UTC
(In reply to qing.wang from comment #86)
> Tests on new qemu version:
> qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64
> Guest win2019
> 
> For virtio-scsi, there are some issues:
> 
> /usr/libexec/qemu-kvm \
>     -name 'avocado-vt-vm1' \
>     -machine q35  \
>     -nodefaults \
>     -device VGA,bus=pcie.0,addr=0x1  \
>     -device
> pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
>     -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
>     -object iothread,id=iothread0 \
>     -device
> pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
>     -device
> virtio-scsi-pci,id=virtio_scsi_pci,iothread=iothread0,bus=pcie.0-root-port-3,
> addr=0x0 \
>     -drive
> id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,
> file=/home/kvm_autotest_root/images/win2019-64-virtio-scsi.qcow2 \
>     -device scsi-hd,id=image1,drive=drive_image1,bootindex=0 \
>     -device
> pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
>     -device
> virtio-net-pci,mac=9a:ab:fa:09:62:1c,id=idLVRSKI,netdev=idfMN0PL,bus=pcie.0-
> root-port-4,addr=0x0  \
>     -netdev tap,id=idfMN0PL,vhost=on \
>     -m 8G  \
>     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
>     -vnc :6  \
>     -qmp tcp:0:5956,server,nowait \
>     -rtc base=localtime,clock=host,driftfix=slew  \
>     -boot order=cdn,once=c,menu=off,strict=off \
>     -enable-kvm \
>     -monitor stdio \
>     -blockdev
> node-name=file_stg1,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg1.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg1,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg1 \
>     -blockdev
> node-name=file_stg2,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg2.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg2,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg2 \
>     \
>     -blockdev
> node-name=file_stg3,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg3.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg3,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg3 \
>     -blockdev
> node-name=file_stg4,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg4.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg4,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg4 \
>     \
>     -blockdev
> node-name=file_stg5,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg5.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg5,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg5 \
>     -blockdev
> node-name=file_stg6,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg6.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg6,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg6 \
>     \
>     -blockdev
> node-name=file_stg7,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg7.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg7,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg7 \
>     -blockdev
> node-name=file_stg8,driver=file,aio=threads,filename=/home/kvm_autotest_root/
> images/stg8.qcow2,cache.direct=on,cache.no-flush=off \
>     -blockdev
> node-name=drive_stg8,driver=qcow2,cache.direct=on,cache.no-flush=off,
> file=file_stg8 \
>     \
>     -device
> pcie-root-port,id=pcie_port_1,slot=5,chassis=5,addr=0x5,bus=pcie.0,
> hotplug=off \
>     -device virtio-scsi-pci,id=scsi1,bus=pcie_port_1,addr=0x0,hotplug=off \
>     -device scsi-hd,id=stg1,drive=drive_stg1,write-cache=on,bus=scsi1.0 \
>     \
>     -device
> pcie-root-port,id=pcie_port_2,slot=6,chassis=6,addr=0x6,bus=pcie.0,
> hotplug=off \
>     -device virtio-scsi-pci,id=scsi2,bus=pcie_port_2,addr=0x0,hotplug=on \
>     -device scsi-hd,id=stg2,drive=drive_stg2,write-cache=on,bus=scsi2.0 \
>     \
>     -device
> pcie-root-port,id=pcie_port_3,slot=7,chassis=7,addr=0x7,bus=pcie.0,
> hotplug=on \
>     -device virtio-scsi-pci,id=scsi3,bus=pcie_port_3,addr=0x0,hotplug=off \
>     -device scsi-hd,id=stg3,drive=drive_stg3,write-cache=on,bus=scsi3.0 \
>     \
>     -device
> pcie-root-port,id=pcie_port_4,slot=8,chassis=8,addr=0x8,bus=pcie.0,
> hotplug=on \
>     -device virtio-scsi-pci,id=scsi4,bus=pcie_port_4,addr=0x0,hotplug=on \
>     -device scsi-hd,id=stg4,drive=drive_stg4,write-cache=on,bus=scsi4.0 \
>     \
>     -device
> pcie-root-port,id=pcie_port_5,slot=9,chassis=9,addr=0x9,bus=pcie.0,
> hotplug=off \
>     \
>     -device
> pcie-root-port,id=pcie_port_6,slot=10,chassis=10,addr=0x10,bus=pcie.0,
> hotplug=off \
>     \
>     -device
> pcie-root-port,id=pcie_port_7,slot=11,chassis=11,addr=0x11,bus=pcie.0 \
>     \
>     -device
> pcie-root-port,id=pcie_port_8,slot=12,chassis=12,addr=0x12,bus=pcie.0 \
> 
> There are 4 disks D(1G) E(2G) F(3G) G(4G) in guest.
> 
> D E not in list,it is good.
> FG are list, i think just G in the list.
> And if we try to remove them (F or G), It is disappear in list, but it exist
> in explorer. And the application become non responding like as file explorer
> and disk management.

If this happens when hotplug=on on the root port, then it's not related to this feature's changes, and should be tracked separately.

Comment 92 Lei Yang 2020-05-18 09:51:52 UTC
(In reply to Julia Suvorova from comment #90)
> (In reply to Lei Yang from comment #88)
> > Hi, Julia
> > 
> > I test params "hotplug=off" on qemu version
> > qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64
> > Guest RHEL.8.2
> > 
> > For virtual network has some issue.
> > QEMU CLI:
> > /usr/libexec/qemu-kvm \
> > -name 'avocado-vt-vm1'  \
> > -sandbox on  \
> > -machine q35 \
> > -device
> > pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,
> > chassis=1 \
> > -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
> > -nodefaults \
> > -device VGA,bus=pcie.0,addr=0x2 \
> > -m 6144  \
> > -smp 6,maxcpus=6,cores=3,threads=1,dies=1,sockets=2  \
> > -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
> > -device
> > pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,
> > chassis=2 \
> > -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
> > -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
> > -device
> > pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,
> > chassis=3 \
> > -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
> > -blockdev
> > node-name=file_image1,driver=file,aio=threads,filename=/home/
> > kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2,cache.direct=on,cache.
> > no-flush=off \
> > -blockdev
> > node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,
> > file=file_image1 \
> > -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
> > -device
> > pcie-root-port,id=pcie-root-port-3,hotplug=off,port=0x3,addr=0x1.0x3,
> > bus=pcie.0,chassis=4 \
> > -device
> > virtio-net-pci,mac=9a:82:59:0b:01:17,id=idVn2ph4,mq=on,vectors=10,
> > netdev=idb8GVnn,bus=pcie-root-port-3,addr=0x0  \
> > -netdev tap,id=idb8GVnn,vhost=on \
> > -vnc :0  \
> > -rtc base=utc,clock=host,driftfix=slew  \
> > -boot menu=off,order=cdn,once=c,strict=off \
> > -enable-kvm \
> > -device
> > pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,
> > addr=0x3,chassis=5 \
> > -monitor stdio \
> > -qmp tcp:0:5555,server,nowait \
> > 
> > ===>steps
> > 1. Test inside the guest.
> > # lspci |grep Eth
> > 05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
> > # echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/remove
> > 
> > 2. The nic has been removed.
> > # ifconfig
> > lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
> >         inet 127.0.0.1  netmask 255.0.0.0
> >         inet6 ::1  prefixlen 128  scopeid 0x10<host>
> >         loop  txqueuelen 1000  (Local Loopback)
> >         RX packets 60  bytes 5336 (5.2 KiB)
> >         RX errors 0  dropped 0  overruns 0  frame 0
> >         TX packets 60  bytes 5336 (5.2 KiB)
> >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > 
> > virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
> >         inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
> >         ether 52:54:00:fe:a5:ed  txqueuelen 1000  (Ethernet)
> >         RX packets 0  bytes 0 (0.0 B)
> >         RX errors 0  dropped 0  overruns 0  frame 0
> >         TX packets 0  bytes 0 (0.0 B)
> >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > 
> > This is not expected result.
> 
> 'echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/remove' nor the same command
> with 'hotplug=on' device will lead to removing device from QEMU (no
> DEVICE_DELETED event).
> This command removes a device only inside Linux, and you can see it back if
> you rescan: 'echo 1 > /sys/bus/pci/rescan'.
> This behaviour is correct.
> 
> If you want to actually remove a device, you need to power down the slot
> that your device inserted to:
> 'echo 0 > /sys/bus/pci/slots/`slot_number`/power'

Hi, Julia

Thanks for your reply,according to the command your provided to me, I did the following test:

==> hotplug = on
-device pcie-root-port,id=pcie-root-port-3,hotplug=on,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,mac=9a:82:59:0b:01:17,id=idVn2ph4,mq=on,vectors=10,netdev=idb8GVnn,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idb8GVnn,vhost=on \

1. check the virtio-net device slot_num
# lspci -vvv -s 0000:05:00.0
05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
	Subsystem: Red Hat, Inc. Device 1100
	Physical Slot: 0-3
2. power down the slot and check the device status
# cat /sys/bus/pci/slots/0-3/power 
1
# echo 0 > /sys/bus/pci/slots/0-3/power
# ip -d link show eth0  (the device has been hot unplug)
Device "eth0" does not exist.

==> hotplug = off
-device pcie-root-port,id=pcie-root-port-3,hotplug=off,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device virtio-net-pci,mac=9a:82:59:0b:01:17,id=idVn2ph4,mq=on,vectors=10,netdev=idb8GVnn,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idb8GVnn,vhost=on \

1. check the virtio-net device slot_num (when hotplug=off, can not check physical slot number)
# lspci -vvv -s 0000:05:00.0

The above is my test result. if it does not match expected result,please tell me know. I will test it again.

Best regards
LeiYang

Comment 93 Julia Suvorova 2020-05-18 10:14:59 UTC
(In reply to Lei Yang from comment #92)
> (In reply to Julia Suvorova from comment #90)
> > (In reply to Lei Yang from comment #88)
> > > Hi, Julia
> > > 
> > > I test params "hotplug=off" on qemu version
> > > qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64
> > > Guest RHEL.8.2
> > > 
> > > For virtual network has some issue.
> > > QEMU CLI:
> > > /usr/libexec/qemu-kvm \
> > > -name 'avocado-vt-vm1'  \
> > > -sandbox on  \
> > > -machine q35 \
> > > -device
> > > pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,
> > > chassis=1 \
> > > -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
> > > -nodefaults \
> > > -device VGA,bus=pcie.0,addr=0x2 \
> > > -m 6144  \
> > > -smp 6,maxcpus=6,cores=3,threads=1,dies=1,sockets=2  \
> > > -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
> > > -device
> > > pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,
> > > chassis=2 \
> > > -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
> > > -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
> > > -device
> > > pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,
> > > chassis=3 \
> > > -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
> > > -blockdev
> > > node-name=file_image1,driver=file,aio=threads,filename=/home/
> > > kvm_autotest_root/images/rhel820-64-virtio-scsi.qcow2,cache.direct=on,cache.
> > > no-flush=off \
> > > -blockdev
> > > node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,
> > > file=file_image1 \
> > > -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
> > > -device
> > > pcie-root-port,id=pcie-root-port-3,hotplug=off,port=0x3,addr=0x1.0x3,
> > > bus=pcie.0,chassis=4 \
> > > -device
> > > virtio-net-pci,mac=9a:82:59:0b:01:17,id=idVn2ph4,mq=on,vectors=10,
> > > netdev=idb8GVnn,bus=pcie-root-port-3,addr=0x0  \
> > > -netdev tap,id=idb8GVnn,vhost=on \
> > > -vnc :0  \
> > > -rtc base=utc,clock=host,driftfix=slew  \
> > > -boot menu=off,order=cdn,once=c,strict=off \
> > > -enable-kvm \
> > > -device
> > > pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,
> > > addr=0x3,chassis=5 \
> > > -monitor stdio \
> > > -qmp tcp:0:5555,server,nowait \
> > > 
> > > ===>steps
> > > 1. Test inside the guest.
> > > # lspci |grep Eth
> > > 05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
> > > # echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/remove
> > > 
> > > 2. The nic has been removed.
> > > # ifconfig
> > > lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
> > >         inet 127.0.0.1  netmask 255.0.0.0
> > >         inet6 ::1  prefixlen 128  scopeid 0x10<host>
> > >         loop  txqueuelen 1000  (Local Loopback)
> > >         RX packets 60  bytes 5336 (5.2 KiB)
> > >         RX errors 0  dropped 0  overruns 0  frame 0
> > >         TX packets 60  bytes 5336 (5.2 KiB)
> > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > 
> > > virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
> > >         inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
> > >         ether 52:54:00:fe:a5:ed  txqueuelen 1000  (Ethernet)
> > >         RX packets 0  bytes 0 (0.0 B)
> > >         RX errors 0  dropped 0  overruns 0  frame 0
> > >         TX packets 0  bytes 0 (0.0 B)
> > >         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> > > 
> > > This is not expected result.
> > 
> > 'echo 1 > /sys/bus/pci/devices/0000\:05\:00.0/remove' nor the same command
> > with 'hotplug=on' device will lead to removing device from QEMU (no
> > DEVICE_DELETED event).
> > This command removes a device only inside Linux, and you can see it back if
> > you rescan: 'echo 1 > /sys/bus/pci/rescan'.
> > This behaviour is correct.
> > 
> > If you want to actually remove a device, you need to power down the slot
> > that your device inserted to:
> > 'echo 0 > /sys/bus/pci/slots/`slot_number`/power'
> 
> Hi, Julia
> 
> Thanks for your reply,according to the command your provided to me, I did
> the following test:
> 
> ==> hotplug = on
> -device
> pcie-root-port,id=pcie-root-port-3,hotplug=on,port=0x3,addr=0x1.0x3,bus=pcie.
> 0,chassis=4 \
> -device
> virtio-net-pci,mac=9a:82:59:0b:01:17,id=idVn2ph4,mq=on,vectors=10,
> netdev=idb8GVnn,bus=pcie-root-port-3,addr=0x0  \
> -netdev tap,id=idb8GVnn,vhost=on \
> 
> 1. check the virtio-net device slot_num
> # lspci -vvv -s 0000:05:00.0
> 05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
> 	Subsystem: Red Hat, Inc. Device 1100
> 	Physical Slot: 0-3
> 2. power down the slot and check the device status
> # cat /sys/bus/pci/slots/0-3/power 
> 1
> # echo 0 > /sys/bus/pci/slots/0-3/power
> # ip -d link show eth0  (the device has been hot unplug)
> Device "eth0" does not exist.
> 
> ==> hotplug = off
> -device
> pcie-root-port,id=pcie-root-port-3,hotplug=off,port=0x3,addr=0x1.0x3,
> bus=pcie.0,chassis=4 \
> -device
> virtio-net-pci,mac=9a:82:59:0b:01:17,id=idVn2ph4,mq=on,vectors=10,
> netdev=idb8GVnn,bus=pcie-root-port-3,addr=0x0  \
> -netdev tap,id=idb8GVnn,vhost=on \
> 
> 1. check the virtio-net device slot_num (when hotplug=off, can not check
> physical slot number)
> # lspci -vvv -s 0000:05:00.0
> 
> The above is my test result. if it does not match expected result,please
> tell me know. I will test it again.

Now you can't locate the slot, this is correct.

Best regards, Julia Suvorova.

Comment 94 Yumei Huang 2020-05-20 10:30:38 UTC
Tried with virtio balloon device, test pass. If set hotplug=off, there is no 'Physical Slot' info in lspci for the device.

Comment 95 Yanghang Liu 2020-05-20 10:59:48 UTC
Hi Julia, 

According to the comment 90 - comment 93, I did the hotunplug test about the vfio-pf/vfio-vf device in RHEL82 vm.


The test result is as following:

1.the pf/vf  in  pcie-root-port with hotplug=off
The  physical slot number of vf/pf device can not be observed  in/sys/bus/pci/slots/$physical_slot/ dir.

2. the pf/vf  in  pcie-root-port with hotplug=on
The physical slot number of vf/pf device can be observed in/sys/bus/pci/slots/$physical_slot/ dir
The vf/pf device can be hot-unplugged successfully by running "echo 0 > /sys/bus/pci/slots/$physical_slot/power" in vm


Can you help me check to see if the test result I got is the expected result?

Comment 96 Julia Suvorova 2020-05-20 16:27:56 UTC
(In reply to yanghliu from comment #95)
> Hi Julia, 
> 
> According to the comment 90 - comment 93, I did the hotunplug test about the
> vfio-pf/vfio-vf device in RHEL82 vm.
> 
> 
> The test result is as following:
> 
> 1.the pf/vf  in  pcie-root-port with hotplug=off
> The  physical slot number of vf/pf device can not be observed 
> in/sys/bus/pci/slots/$physical_slot/ dir.
> 
> 2. the pf/vf  in  pcie-root-port with hotplug=on
> The physical slot number of vf/pf device can be observed
> in/sys/bus/pci/slots/$physical_slot/ dir
> The vf/pf device can be hot-unplugged successfully by running "echo 0 >
> /sys/bus/pci/slots/$physical_slot/power" in vm
> 
> 
> Can you help me check to see if the test result I got is the expected result?

Looks correct.

Comment 97 Qinghua Cheng 2020-05-27 07:19:14 UTC
Verified with vsock device on qemu-kvm build qemu-kvm-4.2.0-21.module+el8.2.1+6586+8b7713b9.x86_64.

qemu-kvm cmd:

...
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x6,chassis=5,hotplug=on \
-device vhost-vsock-pci,id=vhost-vsock-pci0,bus=pci.6,guest-cid=3 \
...

Test inside guest:
# lspci
...
06:00.0 Communication controller: Red Hat, Inc. Device 1053 (rev 01) 

# lspci -vvv -s 0000:06:00.0
06:00.0 Communication controller: Red Hat, Inc. Device 1053 (rev 01)
	Subsystem: Red Hat, Inc. Device 1100
	Physical Slot: 0-4

And the device can be hot unplugged by echo 0 > /sys/bus/pci/slots/0-4/power

When hotplug = off test inside guest:

# lspci
...
06:00.0 Communication controller: Red Hat, Inc. Device 1053 (rev 01) 

# lspci -vvv -s 0000:06:00.0

Can not get device physical slot information, so can not hot unplug the device. Test is PASS with vsock device.

Comment 98 Lei Yang 2020-05-27 09:33:31 UTC
Test e1000e device on qemu-kvm-4.2.0-20.module+el8.2.1+6467+49dc3278.x86_64.
==> hotplug = on
-device pcie-root-port,id=pcie-root-port-3,hotplug=on,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device e1000e,mac=9a:5e:8f:e1:c2:ea,id=idnbHvg3,netdev=idSmMPmp,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idSmMPmp  \

1. check the e1000e device slot_num
# lspci -vvv -s 0000:05:00.0
05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
	Subsystem: Red Hat, Inc. Device 1100
	Physical Slot: 0-3
2. power down the slot and check the device status
# cat /sys/bus/pci/slots/0-3/power 
1
# echo 0 > /sys/bus/pci/slots/0-3/power
# ip -d link show eth0  (the device has been hot unplug)
Device "eth0" does not exist.

==> hotplug = off
-device pcie-root-port,id=pcie-root-port-3,hotplug=off,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
-device e1000e,mac=9a:5e:8f:e1:c2:ea,id=idnbHvg3,netdev=idSmMPmp,bus=pcie-root-port-3,addr=0x0  \
-netdev tap,id=idSmMPmp  \

1. check the e1000e device slot_num (when hotplug=off, can not check physical slot number)
# lspci -vvv -s 0000:05:00.0

Test pass.

Comment 101 errata-xmlrpc 2020-07-28 07:12:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3172


Note You need to log in before you can comment on or make changes to this bug.