After hot-plug the pcie pci and disk, we can see the hot-plug devices in the dumpxml, but can't see them in the guest, so clone the qemu exsited bug to libvirt to tracking this issue.

Details please see the following 2 comments:
hot-plug the pcie interface:
https://bugzilla.redhat.com/show_bug.cgi?id=819968#c18
hot-plug the pcie disk:
https://bugzilla.redhat.com/show_bug.cgi?id=819968#c19

Two notes about this:

1) the problem described in the original report was due to operator error, and is no longer an issue. The devices that were being plugged into the pcie.0 bus aren't intended to be plugged into a pcie bus, but into a pci bus; libvirt will automatically add a pci-bridge controller and plug those devices into the pci-bridge controller.

2) In terms of supporting hotplug of actual PCIe devices, it's unclear just how important that is, since afaik currently every emulated device available in qemu (except perhaps the pcie-specific controllers listed below) can be plugged into a PCI bus, and most *shouldn't* be plugged into a PCIe bus (although they all allow it)

If we do really need to support hot-plug into a PCIe controller, then (part of) what libvirt needs is to add support for the following pci controller types:

<controller type='pci' model='root-port'/>
<controller type='pci' model='upstream-switch-port'>
<controller type='pci' model='downstream-switch-port'>

The following comment describes what controllers/devices these can connect to:

  https://bugzilla.redhat.com/show_bug.cgi?id=819968#c6

The upstream commit that adds support for the "pci-root" controller is a useful example of what needs to be changed to support a new PCI controller type.

Verify this bug with libvirt-1.2.8-10.el7.x86_64 and qemu-kvm-rhev-2.1.2-16.el7.x86_64.


Scenario 1: Hot-plug a disk device to guest. The disk can't be find in guest.

1. Hot-plug a disk device to guest.

On host:

[root@localhost ~]# cat pcie.xml 
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/pcie.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </disk>

[root@localhost ~]# virsh start 7
Domain 7 started

[root@localhost ~]# virsh attach-device 7 pcie.xml 
Device attached successfully

[root@localhost ~]# virsh dumpxml 7
...
    <type arch='x86_64' machine='pc-q35-rhel7.1.0'>hvm</type>
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/r7.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/pcie.img'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
...


2. Check the disk in guest.

[root@localhost ~]# lspci -t -vvv
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0  Intel Corporation 82801I (ICH9 Family) HD Audio Controller
           +-05.0-[01-02]----01.0-[02]--+-01.0  Red Hat, Inc Virtio network device
           |                            +-02.0  Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II]
           |                            +-04.0  Red Hat, Inc Virtio console
           |                            +-05.0  Red Hat, Inc Virtio block device
           |                            \-06.0  Red Hat, Inc Virtio memory balloon
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

[root@localhost ~]# fdisk -l

Disk /dev/vda: 9663 MB, 9663676416 bytes, 18874368 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0009764c

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    18874367     8924160   8e  Linux LVM

Disk /dev/mapper/rhel-root: 8166 MB, 8166309888 bytes, 15949824 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/rhel-swap: 968 MB, 968884224 bytes, 1892352 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


3. Hot-unplug the disk.
[root@localhost ~]# virsh detach-device 7 pcie.xml 
Device detached successfully



Scenario 2: Cold-plug a disk device to guest. The disk can read and write in guest.

On host:

[root@localhost ~]# virsh start 7
Domain 7 started

[root@localhost ~]# virsh dumpxml 7
...
    <type arch='x86_64' machine='pc-q35-rhel7.1.0'>hvm</type>
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/r7.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/pcie.img'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
...


On guest:

[root@localhost ~]# lspci -t -vvv
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-02.0  Intel Corporation 82801I (ICH9 Family) HD Audio Controller
           +-05.0-[01-02]----01.0-[02]--+-01.0  Red Hat, Inc Virtio network device
           |                            +-02.0  Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II]
           |                            +-04.0  Red Hat, Inc Virtio console
           |                            +-05.0  Red Hat, Inc Virtio block device
           |                            +-06.0  Red Hat, Inc Virtio memory balloon
           |                            \-07.0  Red Hat, Inc Virtio block device
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

[root@localhost ~]# fdisk -l

Disk /dev/vda: 9663 MB, 9663676416 bytes, 18874368 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0009764c

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048     1026047      512000   83  Linux
/dev/vda2         1026048    18874367     8924160   8e  Linux LVM

Disk /dev/vdb: 10 MB, 10485760 bytes, 20480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xf85a03bc

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1            2048       20479        9216   83  Linux

Disk /dev/mapper/rhel-root: 8166 MB, 8166309888 bytes, 15949824 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/rhel-swap: 968 MB, 968884224 bytes, 1892352 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@localhost ~]# mount /dev/vdb1 /mnt
[root@localhost ~]# cd /mnt/
[root@localhost mnt]# ls
file  lost+found
[root@localhost mnt]# touch newfile
[root@localhost mnt]# ls
file  lost+found  newfile

Test this bug with latest libvirt/qemu(qemu-kvm-rhev-2.3.0-18.el7.x86_64,libvirt-1.2.17-6.el7.x86_64) just as step in comment 9, libvirt can hotplug device to pci-brige while qemu still doesn't support, this issuse depend on bug 953008 which still be in ASSIGNED status; moreover, there is bug 1207751 for PCIe support still be in post status. So I will do more test after these two bug got fixed.

Unless you are specifically needing to plug into a pcie-root-port or pcie-switch-downstream-port (which would only be needed for AER, which anyway isn't supported yet in qemu), Bug 1207751 is completely irrelevant. The patches for that bug just add the following new PCI controllers:

   pcie-root-port
   pcie-switch-upstream-port
   pcie-switch-downstream-port

The addition of those controllers has no effect on whether or not the pci-bridge controller is able to accept hot-plugged devices.

Hi, 

The dependent bz953008 is deferred to RHEL7.4. Can we move it to RHEL7.4 as well?

Yes. Also, the description of this bug is grossly out of date and incorrect. The originally-reported problem (unable to hotplug to pcie.0) was not a bug and is expected behavior. The original bug's description has been changed to this:

[Q35] PCI hot plug of devices under legacy pci-bridge with '-M q35' does not work

and that is what is now being tracked by Bug 953008, so I'm changing the description of this BZ to be the same.

Verified on:
libvirt-3.2.0-2.el7.x86_64
qemu-kvm-rhev-2.8.0-6.el7.x86_64
Guest kernel:
kernel-3.10.0-514.el7.x86_64

1. Prepare a q35 VM with pcie-root-port pcie-switch-upstream-port pcie-switch-downstream-port like following:
 <controller type='pci' index='6' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='6' port='0x14'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='7' model='pcie-switch-upstream-port'>
      <model name='x3130-upstream'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='8' model='pcie-switch-downstream-port'>
      <model name='xio3130-downstream'/>
      <target chassis='8' port='0x0'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </controller>
2. Create two raw file and attach to the pcie-root-port pcie-switch-downstream-port. 
# virsh attach-disk q35-upstream /tmp/vdb vdb --address pci:0000.06.00.0
Disk attached successfully

# virsh attach-disk q35-upstream /tmp/vdb vdc --address pci:0000.08.00.0
Disk attached successfully

3. Check the disks in VM:
# lsblk
NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda             8:0    0    10G  0 disk 
├─sda1          8:1    0     1G  0 part /boot
└─sda2          8:2    0     9G  0 part 
  ├─rhel-root 253:0    0     8G  0 lvm  /
  └─rhel-swap 253:1    0     1G  0 lvm  [SWAP]
vda           252:0    0 192.5K  0 disk 
vdb           252:16   0 192.5K  0 disk

I am sorry to misunderstand the title of the bug. Set back to ON_QA and waiting for the fix of BZ953008

According to comment20 and the status of BZ953008, change to ASSIGNED.

Hi jingzhao, the hotplug/hotunplug seems supported now.
I tried on libvirt-3.9.0-7.el7.x86_64 qemu-kvm-rhev-2.10.0-14.el7.x86_64 and it worked:
Steps:
1. Prepare a running q35 VM with following xml:
```
  <os>
    <type arch='x86_64' machine='pc-q35-rhel7.5.0'>hvm</type>
    <boot dev='hd'/>
  </os>
...
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2' index='9'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
```
2. Attach a disk to the pci-bridge
# virsh attach-disk q35 /tmp/aa vdb --address pci:0000.02.a.0
Disk attached successfully

Check disk in VM;
(VM)# lsblk /dev/vdb
NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
vdb  252:16   0 192.5K  0 disk

I find the pci-bridge hotplug-support in qemu-2.10's changelog(https://wiki.qemu.org/ChangeLog/2.10#PCI.2FPCIe). I am not sure if that makes it work because the BZ953008 is not fixed.
Please check if the pci-bridge hotplug in q35 machine is supported in qemu now.

The needinfo of comment23

(In reply to Han Han from comment #24)
> The needinfo of comment23

Test against with qemu-kvm-rhev-2.10.0-15.el7.x86_64 and also hit the issue

1. Boot guest with qemu command line [1]
/usr/libexec/qemu-kvm \
-M q35 \
-cpu Haswell-noTSX \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-global isa-debugcon.iobase=0x402 \
-boot menu=on \
-qmp tcp:0:6667,server,nowait \
-usb \
-device usb-tablet \
-vga qxl \
-device pcie-root-port,bus=pcie.0,id=root0,multifunction=on,chassis=1,addr=0xa.0 \
-drive file=/home/sriov/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=root0 \
-device pcie-root-port,bus=pcie.0,id=root1,chassis=11,addr=0xa.1 \
-device virtio-net-pci,netdev=tap10,mac=00:52:68:26:31:03,bus=root1,id=net0 -netdev tap,id=tap10 \
-device pcie-root-port,bus=pcie.0,id=root2,chassis=12,addr=0xa.2 \
-device virtio-net-pci,netdev=tap11,mac=00:52:68:26:31:00,bus=root2,id=net1 -netdev tap,id=tap11 \
-device pcie-root-port,bus=pcie.0,id=root6,chassis=15,addr=0xa.5 \
-device vfio-pci,host=04:02.0,id=vf1,bus=root6,x-req=off \
-device pcie-root-port,bus=pcie.0,id=root7,chassis=4 \
-device vfio-pci,host=04:02.1,id=vf2,bus=root7,x-req=off \
-device pci-bridge,id=bridge0,chassis_nr=2 \
-monitor stdio \
-vnc :1 \

2. hot plug disk to pci-bridge
{"execute":"__com.redhat_drive_add", "arguments": {"file":"/home/sriov/block1.qcow2","format":"qcow2","id":"test1"}}
{"return": {}}
{"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test1","id":"test1","bus":"bridge0","addr":"0x1"}}
{"return": {}}

Result: didn't find the disk in guest

Thanks
Jing

Created attachment 1440392 [details]
The libvirt log of hot-plug