Red Hat Bugzilla – Bug 1377160
[RFE] Q35: Implement hotplug for pxb-pcie devices
Last modified: 2017-08-01 23:29:59 EDT
Description of problem: When hotplugging to a pcie-root-port connected to a pcie-expander-bus (i.e. pcie-pxb --> pcie-root-port), the hotplug is successful but the guest doesn't see the new device. See the similar problem when hot-plugging to pcie-switch-downstream-port connected to pcie-expander-bus (i.e. pcie-pxb --> pcie-root-port --> pcie-switch-upstream-port --> pcie-switch-downstream-port) Version-Release number of selected component (if applicable): libvirt-2.0.0-9.el7.x86_64 qemu-kvm-rhev-2.6.0-25.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. start vm with pcie-pxb --> pcie-root-port <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <model name='i82801b11-bridge'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <model name='pci-bridge'/> <target chassisNr='2'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-expander-bus'> <model name='pxb-pcie'/> <target busNr='100'> <node>1</node> </target> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='ioh3420'/> <target chassis='4' port='0x0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <interface type='direct'> <mac address='52:54:00:65:18:d8'/> <source dev='eno1' mode='bridge'/> <target dev='macvtap0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> 2. hot-plug nic into root-port # cat nic.xml <interface type='network'> <mac address='52:54:00:3a:81:9a'/> <source network='default' bridge='virbr0'/> <model type='e1000'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> # virsh attach-device vm1-q35-Sec nic.xml Device attached successfully check domain xml, there are 2 nics # virsh dumpxml vm1-q35-Sec | grep inter -a6 <interface type='direct'> <mac address='52:54:00:65:18:d8'/> <source dev='eno1' mode='bridge'/> <target dev='macvtap0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:3a:81:9a'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='e1000'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> checked in guest, only 1 nic is found. Guest cannot see the hot-plugged nic # ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.66.4.171 netmask 255.255.252.0 broadcast 10.66.7.255 inet6 fe80::5054:ff:fe65:18d8 prefixlen 64 scopeid 0x20<link> ether 52:54:00:65:18:d8 txqueuelen 1000 (Ethernet) RX packets 133 bytes 10172 (9.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 15 bytes 1986 (1.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 3. hot-unplug the nic # virsh detach-device vm1-q35-Sec nic.xml Device detached successfully 4. checked in domain xml. The nic xml still exists in domain xml # virsh dumpxml vm1-q35-Sec | grep inter -a6 <interface type='direct'> <mac address='52:54:00:65:18:d8'/> <source dev='eno1' mode='bridge'/> <target dev='macvtap0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> </interface> <interface type='network'> <mac address='52:54:00:3a:81:9a'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='e1000'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> Actual results: When hot-plugging to root-port(or downstream-port) connected to pcie-pxb, hotplug is successful but guest cannot see the new device Expected results: When hot-plugging to root-port(or downstream-port) connected to pcie-pxb, hotplug is successful and guest can see the new device Additional info:
This seems like something that should work for Q35. Is it a bug or missing functionality?
(In reply to Karen Noel from comment #3) > This seems like something that should work for Q35. Is it a bug or missing > functionality? Hot-plugging/unplugging for devices attached to Root Ports/Downstream ports of the main host bridge (Root Complex) works OK. The hot-plug support for extra host bridges (pxb/pxb-pcie) is not implemented yet. Thanks, Marcel
Reproduce the bz on qemu-kvm-rhev-2.8.0-6.el7.x86_64 Verified the bz on qemu-kvm-rhev-2.9.0-1.el7.x86_64 Following are the detailed info 1. Boot guest with qemu command line [1] 2. Hot-plug nic device with qmp command: {"execute": "netdev_add", "arguments": { "type":"tap","id":"net1"}} {"return": {}} {"execute": "device_add", "arguments": { "driver":"virtio-net-pci","netdev":"net1","mac":"22:11:22:45:61:98","id":"net1","bus":"root2"}} {"return": {}} 3. In guest, check the nic device works well [root@localhost ~]# lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04) 00:02.0 Host bridge: Red Hat, Inc. Device 000b 00:03.0 Host bridge: Red Hat, Inc. Device 000b 00:04.0 Host bridge: Red Hat, Inc. Device 000b 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 08:00.0 PCI bridge: Red Hat, Inc. Device 000c 09:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01) 14:00.0 PCI bridge: Red Hat, Inc. Device 000c 15:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01) 28:00.0 PCI bridge: Red Hat, Inc. Device 000c 29:00.0 Ethernet controller: Red Hat, Inc Virtio network device (rev 01) [root@localhost ~]# lspci -vvv -t -+-[0000:28]---00.0-[29]----00.0 Red Hat, Inc Virtio network device +-[0000:14]---00.0-[15]----00.0 Red Hat, Inc Virtio block device +-[0000:08]---00.0-[09]----00.0 Red Hat, Inc Virtio block device \-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller +-01.0 Red Hat, Inc. QXL paravirtual graphic card +-02.0 Red Hat, Inc. Device 000b +-03.0 Red Hat, Inc. Device 000b +-04.0 Red Hat, Inc. Device 000b +-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller +-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] \-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller [root@localhost ~]# cat /sys/bus/pci/devices/0000\:09\:00.0/numa_node 0 [root@localhost ~]# cat /sys/bus/pci/devices/0000\:15\:00.0/numa_node 1 [root@localhost ~]# cat /sys/bus/pci/devices/0000\:29\:00.0/numa_node 2 [root@localhost ~]# ping 10.66.4.211 PING 10.66.4.211 (10.66.4.211) 56(84) bytes of data. 64 bytes from 10.66.4.211: icmp_seq=1 ttl=64 time=1.76 ms 64 bytes from 10.66.4.211: icmp_seq=2 ttl=64 time=0.925 ms ^C --- 10.66.4.211 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.925/1.345/1.765/0.420 ms 4. Unplug the nic device in hmp (qemu) netdev_del net1 (qemu) device_del net1 5. Chekc the nic in guest, and nic device can be deleted [1] /usr/libexec/qemu-kvm \ -machine q35,smm=on,accel=kvm \ -cpu Haswell-noTSX \ -nodefaults -rtc base=utc \ -m 4G \ -smp 4,sockets=4,cores=1,threads=1 \ -object memory-backend-ram,size=1024M,id=ram-node0 \ -numa node,nodeid=0,cpus=0,memdev=ram-node0 \ -object memory-backend-ram,size=1024M,id=ram-node1 \ -numa node,nodeid=1,cpus=1,memdev=ram-node1 \ -object memory-backend-ram,size=2048M,id=ram-node2 \ -numa node,nodeid=2,cpus=2,memdev=ram-node2 \ -enable-kvm \ -uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \ -k en-us \ -nodefaults \ -serial unix:/tmp/serial0,server,nowait \ -boot menu=on \ -qmp tcp:0:6666,server,nowait \ -vga qxl \ -chardev file,path=/home/seabios.log,id=seabios -device isa-debugcon,chardev=seabios,iobase=0x402 \ -device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \ -device pcie-root-port,bus=bridge1,id=root0,slot=1 \ -drive file=/home/test/rhel/rhel74.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=root0,bootindex=0 \ -device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=20 \ -device pcie-root-port,bus=bridge2,id=root1,slot=2 \ -drive file=/home/test/rhel/block1.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \ -device virtio-blk-pci,drive=drive-virtio-disk1,id=virtio-disk1,bus=root1 \ -device pxb-pcie,id=bridge3,bus=pcie.0,numa_node=2,bus_nr=40 \ -device pcie-root-port,id=root2,slot=3,bus=bridge3 \ -monitor stdio \ -vnc :0 \ Thanks Jing
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:2392