Bug 1045975
| Summary: | [Q35] PCI hot plug of devices under legacy pci-bridge with '-M q35' does not work | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Xuesong Zhang <xuzhang> | ||||
| Component: | libvirt | Assignee: | Pavel Hrdina <phrdina> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | Meina Li <meili> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.0 | CC: | dyuan, hhan, hhuang, jdenemar, jinzhao, jsuchane, juzhang, knoel, kraxel, laine, lcheng, lmen, marcel, mst, mzhan, nerijus, ovasik, phrdina, qzhang, virt-maint, yafu | ||||
| Target Milestone: | beta | Keywords: | TestOnly | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | 902190 | Environment: | |||||
| Last Closed: | 2018-08-09 03:56:48 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 902190, 953008 | ||||||
| Bug Blocks: | 922007 | ||||||
| Attachments: |
|
||||||
|
Comment 1
Xuesong Zhang
2013-12-23 05:18:12 UTC
Two notes about this: 1) the problem described in the original report was due to operator error, and is no longer an issue. The devices that were being plugged into the pcie.0 bus aren't intended to be plugged into a pcie bus, but into a pci bus; libvirt will automatically add a pci-bridge controller and plug those devices into the pci-bridge controller. 2) In terms of supporting hotplug of actual PCIe devices, it's unclear just how important that is, since afaik currently every emulated device available in qemu (except perhaps the pcie-specific controllers listed below) can be plugged into a PCI bus, and most *shouldn't* be plugged into a PCIe bus (although they all allow it) If we do really need to support hot-plug into a PCIe controller, then (part of) what libvirt needs is to add support for the following pci controller types: <controller type='pci' model='root-port'/> <controller type='pci' model='upstream-switch-port'> <controller type='pci' model='downstream-switch-port'> The following comment describes what controllers/devices these can connect to: https://bugzilla.redhat.com/show_bug.cgi?id=819968#c6 The upstream commit that adds support for the "pci-root" controller is a useful example of what needs to be changed to support a new PCI controller type. Verify this bug with libvirt-1.2.8-10.el7.x86_64 and qemu-kvm-rhev-2.1.2-16.el7.x86_64.
Scenario 1: Hot-plug a disk device to guest. The disk can't be find in guest.
1. Hot-plug a disk device to guest.
On host:
[root@localhost ~]# cat pcie.xml
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/pcie.img'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
</disk>
[root@localhost ~]# virsh start 7
Domain 7 started
[root@localhost ~]# virsh attach-device 7 pcie.xml
Device attached successfully
[root@localhost ~]# virsh dumpxml 7
...
<type arch='x86_64' machine='pc-q35-rhel7.1.0'>hvm</type>
...
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/r7.qcow2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/pcie.img'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
</disk>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
</controller>
...
2. Check the disk in guest.
[root@localhost ~]# lspci -t -vvv
-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
+-01.0 Red Hat, Inc. QXL paravirtual graphic card
+-02.0 Intel Corporation 82801I (ICH9 Family) HD Audio Controller
+-05.0-[01-02]----01.0-[02]--+-01.0 Red Hat, Inc Virtio network device
| +-02.0 Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II]
| +-04.0 Red Hat, Inc Virtio console
| +-05.0 Red Hat, Inc Virtio block device
| \-06.0 Red Hat, Inc Virtio memory balloon
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
+-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
\-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# fdisk -l
Disk /dev/vda: 9663 MB, 9663676416 bytes, 18874368 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0009764c
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 1026047 512000 83 Linux
/dev/vda2 1026048 18874367 8924160 8e Linux LVM
Disk /dev/mapper/rhel-root: 8166 MB, 8166309888 bytes, 15949824 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/rhel-swap: 968 MB, 968884224 bytes, 1892352 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
3. Hot-unplug the disk.
[root@localhost ~]# virsh detach-device 7 pcie.xml
Device detached successfully
Scenario 2: Cold-plug a disk device to guest. The disk can read and write in guest.
On host:
[root@localhost ~]# virsh start 7
Domain 7 started
[root@localhost ~]# virsh dumpxml 7
...
<type arch='x86_64' machine='pc-q35-rhel7.1.0'>hvm</type>
...
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/r7.qcow2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/pcie.img'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
</disk>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
</controller>
...
On guest:
[root@localhost ~]# lspci -t -vvv
-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
+-01.0 Red Hat, Inc. QXL paravirtual graphic card
+-02.0 Intel Corporation 82801I (ICH9 Family) HD Audio Controller
+-05.0-[01-02]----01.0-[02]--+-01.0 Red Hat, Inc Virtio network device
| +-02.0 Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II]
| +-04.0 Red Hat, Inc Virtio console
| +-05.0 Red Hat, Inc Virtio block device
| +-06.0 Red Hat, Inc Virtio memory balloon
| \-07.0 Red Hat, Inc Virtio block device
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
+-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
\-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# fdisk -l
Disk /dev/vda: 9663 MB, 9663676416 bytes, 18874368 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0009764c
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 1026047 512000 83 Linux
/dev/vda2 1026048 18874367 8924160 8e Linux LVM
Disk /dev/vdb: 10 MB, 10485760 bytes, 20480 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xf85a03bc
Device Boot Start End Blocks Id System
/dev/vdb1 2048 20479 9216 83 Linux
Disk /dev/mapper/rhel-root: 8166 MB, 8166309888 bytes, 15949824 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/rhel-swap: 968 MB, 968884224 bytes, 1892352 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@localhost ~]# mount /dev/vdb1 /mnt
[root@localhost ~]# cd /mnt/
[root@localhost mnt]# ls
file lost+found
[root@localhost mnt]# touch newfile
[root@localhost mnt]# ls
file lost+found newfile
Test this bug with latest libvirt/qemu(qemu-kvm-rhev-2.3.0-18.el7.x86_64,libvirt-1.2.17-6.el7.x86_64) just as step in comment 9, libvirt can hotplug device to pci-brige while qemu still doesn't support, this issuse depend on bug 953008 which still be in ASSIGNED status; moreover, there is bug 1207751 for PCIe support still be in post status. So I will do more test after these two bug got fixed. Unless you are specifically needing to plug into a pcie-root-port or pcie-switch-downstream-port (which would only be needed for AER, which anyway isn't supported yet in qemu), Bug 1207751 is completely irrelevant. The patches for that bug just add the following new PCI controllers: pcie-root-port pcie-switch-upstream-port pcie-switch-downstream-port The addition of those controllers has no effect on whether or not the pci-bridge controller is able to accept hot-plugged devices. Hi, The dependent bz953008 is deferred to RHEL7.4. Can we move it to RHEL7.4 as well? Yes. Also, the description of this bug is grossly out of date and incorrect. The originally-reported problem (unable to hotplug to pcie.0) was not a bug and is expected behavior. The original bug's description has been changed to this: [Q35] PCI hot plug of devices under legacy pci-bridge with '-M q35' does not work and that is what is now being tracked by Bug 953008, so I'm changing the description of this BZ to be the same. Verified on:
libvirt-3.2.0-2.el7.x86_64
qemu-kvm-rhev-2.8.0-6.el7.x86_64
Guest kernel:
kernel-3.10.0-514.el7.x86_64
1. Prepare a q35 VM with pcie-root-port pcie-switch-upstream-port pcie-switch-downstream-port like following:
<controller type='pci' index='6' model='pcie-root-port'>
<model name='ioh3420'/>
<target chassis='6' port='0x14'/>
<alias name='pci.6'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='7' model='pcie-switch-upstream-port'>
<model name='x3130-upstream'/>
<alias name='pci.7'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='8' model='pcie-switch-downstream-port'>
<model name='xio3130-downstream'/>
<target chassis='8' port='0x0'/>
<alias name='pci.8'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</controller>
2. Create two raw file and attach to the pcie-root-port pcie-switch-downstream-port.
# virsh attach-disk q35-upstream /tmp/vdb vdb --address pci:0000.06.00.0
Disk attached successfully
# virsh attach-disk q35-upstream /tmp/vdb vdc --address pci:0000.08.00.0
Disk attached successfully
3. Check the disks in VM:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 9G 0 part
├─rhel-root 253:0 0 8G 0 lvm /
└─rhel-swap 253:1 0 1G 0 lvm [SWAP]
vda 252:0 0 192.5K 0 disk
vdb 252:16 0 192.5K 0 disk
I am sorry to misunderstand the title of the bug. Set back to ON_QA and waiting for the fix of BZ953008 Hi jingzhao, the hotplug/hotunplug seems supported now.
I tried on libvirt-3.9.0-7.el7.x86_64 qemu-kvm-rhev-2.10.0-14.el7.x86_64 and it worked:
Steps:
1. Prepare a running q35 VM with following xml:
```
<os>
<type arch='x86_64' machine='pc-q35-rhel7.5.0'>hvm</type>
<boot dev='hd'/>
</os>
...
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2' index='9'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
```
2. Attach a disk to the pci-bridge
# virsh attach-disk q35 /tmp/aa vdb --address pci:0000.02.a.0
Disk attached successfully
Check disk in VM;
(VM)# lsblk /dev/vdb
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vdb 252:16 0 192.5K 0 disk
I find the pci-bridge hotplug-support in qemu-2.10's changelog(https://wiki.qemu.org/ChangeLog/2.10#PCI.2FPCIe). I am not sure if that makes it work because the BZ953008 is not fixed.
Please check if the pci-bridge hotplug in q35 machine is supported in qemu now.
The needinfo of comment23 (In reply to Han Han from comment #24) > The needinfo of comment23 Test against with qemu-kvm-rhev-2.10.0-15.el7.x86_64 and also hit the issue 1. Boot guest with qemu command line [1] /usr/libexec/qemu-kvm \ -M q35 \ -cpu Haswell-noTSX \ -nodefaults -rtc base=utc \ -m 4G \ -smp 4,sockets=4,cores=1,threads=1 \ -enable-kvm \ -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ -k en-us \ -nodefaults \ -global isa-debugcon.iobase=0x402 \ -boot menu=on \ -qmp tcp:0:6667,server,nowait \ -usb \ -device usb-tablet \ -vga qxl \ -device pcie-root-port,bus=pcie.0,id=root0,multifunction=on,chassis=1,addr=0xa.0 \ -drive file=/home/sriov/rhel75-seabios-bk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=root0 \ -device pcie-root-port,bus=pcie.0,id=root1,chassis=11,addr=0xa.1 \ -device virtio-net-pci,netdev=tap10,mac=00:52:68:26:31:03,bus=root1,id=net0 -netdev tap,id=tap10 \ -device pcie-root-port,bus=pcie.0,id=root2,chassis=12,addr=0xa.2 \ -device virtio-net-pci,netdev=tap11,mac=00:52:68:26:31:00,bus=root2,id=net1 -netdev tap,id=tap11 \ -device pcie-root-port,bus=pcie.0,id=root6,chassis=15,addr=0xa.5 \ -device vfio-pci,host=04:02.0,id=vf1,bus=root6,x-req=off \ -device pcie-root-port,bus=pcie.0,id=root7,chassis=4 \ -device vfio-pci,host=04:02.1,id=vf2,bus=root7,x-req=off \ -device pci-bridge,id=bridge0,chassis_nr=2 \ -monitor stdio \ -vnc :1 \ 2. hot plug disk to pci-bridge {"execute":"__com.redhat_drive_add", "arguments": {"file":"/home/sriov/block1.qcow2","format":"qcow2","id":"test1"}} {"return": {}} {"execute":"device_add","arguments":{"driver":"virtio-blk-pci","drive":"test1","id":"test1","bus":"bridge0","addr":"0x1"}} {"return": {}} Result: didn't find the disk in guest Thanks Jing Created attachment 1440392 [details]
The libvirt log of hot-plug
|