Bug 1377160

Summary: [RFE] Q35: Implement hotplug for pxb-pcie devices
Product: Red Hat Enterprise Linux 7 Reporter: Yang Yang <yanyang>
Component: qemu-kvm-rhevAssignee: Marcel Apfelbaum <marcel>
Status: CLOSED ERRATA QA Contact: jingzhao <jinzhao>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: ailan, chayang, dyuan, jinzhao, juzhang, knoel, marcel, mrezanin, virt-maint
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.9.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 23:34:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yang Yang 2016-09-19 02:18:08 UTC 
Description of problem:
When hotplugging to a pcie-root-port connected to a pcie-expander-bus (i.e. pcie-pxb --> pcie-root-port), the hotplug is successful but the guest doesn't see the new device. 
See the similar problem when hot-plugging to pcie-switch-downstream-port connected to pcie-expander-bus (i.e. pcie-pxb --> pcie-root-port --> pcie-switch-upstream-port --> pcie-switch-downstream-port)

Version-Release number of selected component (if applicable):
libvirt-2.0.0-9.el7.x86_64
qemu-kvm-rhev-2.6.0-25.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. start vm with pcie-pxb --> pcie-root-port
<controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='100'>
        <node>1</node>
      </target>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='4' port='0x0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>

2. hot-plug nic into root-port
# cat nic.xml 
<interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

# virsh attach-device vm1-q35-Sec nic.xml 
Device attached successfully

check domain xml, there are 2 nics
# virsh dumpxml vm1-q35-Sec | grep inter -a6
      
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
   
checked in guest, only 1 nic is found. Guest cannot see the hot-plugged nic

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.4.171  netmask 255.255.252.0  broadcast 10.66.7.255
        inet6 fe80::5054:ff:fe65:18d8  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:65:18:d8  txqueuelen 1000  (Ethernet)
        RX packets 133  bytes 10172 (9.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15  bytes 1986 (1.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

3. hot-unplug the nic
# virsh detach-device vm1-q35-Sec nic.xml 
Device detached successfully

4. checked in domain xml. The nic xml still exists in domain xml

# virsh dumpxml vm1-q35-Sec | grep inter -a6
     
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

Actual results:
When hot-plugging to root-port(or downstream-port) connected to pcie-pxb, hotplug is successful but guest cannot see the new device

Expected results:
When hot-plugging to root-port(or downstream-port) connected to pcie-pxb, hotplug is successful and guest can see the new device

Additional info:
Comment 3 Karen Noel 2016-09-21 00:46:11 UTC 
This seems like something that should work for Q35. Is it a bug or missing functionality?
Comment 4 Marcel Apfelbaum 2016-09-28 09:58:51 UTC 
(In reply to Karen Noel from comment #3)
> This seems like something that should work for Q35. Is it a bug or missing
> functionality?

Hot-plugging/unplugging for devices attached to Root Ports/Downstream ports
of the main host bridge (Root Complex) works OK.

The hot-plug support for extra host bridges (pxb/pxb-pcie) is not implemented yet. 


Thanks,
Marcel
Comment 5 jingzhao 2017-04-24 09:10:50 UTC 
Reproduce the bz on qemu-kvm-rhev-2.8.0-6.el7.x86_64

Verified the bz on qemu-kvm-rhev-2.9.0-1.el7.x86_64

Following are the detailed info

1. Boot guest with qemu command line [1]

2. Hot-plug nic device with qmp command:
{"execute": "netdev_add", "arguments": { "type":"tap","id":"net1"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"virtio-net-pci","netdev":"net1","mac":"22:11:22:45:61:98","id":"net1","bus":"root2"}}
{"return": {}}


3. In guest, check the nic device works well
[root@localhost ~]# lspci 
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04)
00:02.0 Host bridge: Red Hat, Inc. Device 000b
00:03.0 Host bridge: Red Hat, Inc. Device 000b
00:04.0 Host bridge: Red Hat, Inc. Device 000b
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
08:00.0 PCI bridge: Red Hat, Inc. Device 000c
09:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
14:00.0 PCI bridge: Red Hat, Inc. Device 000c
15:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
28:00.0 PCI bridge: Red Hat, Inc. Device 000c
29:00.0 Ethernet controller: Red Hat, Inc Virtio network device (rev 01)
[root@localhost ~]# lspci -vvv -t
-+-[0000:28]---00.0-[29]----00.0  Red Hat, Inc Virtio network device
 +-[0000:14]---00.0-[15]----00.0  Red Hat, Inc Virtio block device
 +-[0000:08]---00.0-[09]----00.0  Red Hat, Inc Virtio block device
 \-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
             +-01.0  Red Hat, Inc. QXL paravirtual graphic card
             +-02.0  Red Hat, Inc. Device 000b
             +-03.0  Red Hat, Inc. Device 000b
             +-04.0  Red Hat, Inc. Device 000b
             +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
             +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
             \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:09\:00.0/numa_node 
0
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:15\:00.0/numa_node 
1
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:29\:00.0/numa_node 
2
[root@localhost ~]# ping 10.66.4.211
PING 10.66.4.211 (10.66.4.211) 56(84) bytes of data.
64 bytes from 10.66.4.211: icmp_seq=1 ttl=64 time=1.76 ms
64 bytes from 10.66.4.211: icmp_seq=2 ttl=64 time=0.925 ms
^C
--- 10.66.4.211 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.925/1.345/1.765/0.420 ms

4. Unplug the nic device in hmp
(qemu) netdev_del net1   
(qemu) device_del net1 

5. Chekc the nic in guest, and nic device can be deleted

[1]
/usr/libexec/qemu-kvm \
-machine q35,smm=on,accel=kvm \
-cpu Haswell-noTSX \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-object memory-backend-ram,size=1024M,id=ram-node0 \
-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
-object memory-backend-ram,size=1024M,id=ram-node1 \
-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
-object memory-backend-ram,size=2048M,id=ram-node2 \
-numa node,nodeid=2,cpus=2,memdev=ram-node2 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-serial unix:/tmp/serial0,server,nowait \
-boot menu=on \
-qmp tcp:0:6666,server,nowait \
-vga qxl \
-chardev file,path=/home/seabios.log,id=seabios -device isa-debugcon,chardev=seabios,iobase=0x402 \
-device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \
-device pcie-root-port,bus=bridge1,id=root0,slot=1 \
-drive file=/home/test/rhel/rhel74.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=root0,bootindex=0 \
-device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=20 \
-device pcie-root-port,bus=bridge2,id=root1,slot=2 \
-drive file=/home/test/rhel/block1.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk1,id=virtio-disk1,bus=root1 \
-device pxb-pcie,id=bridge3,bus=pcie.0,numa_node=2,bus_nr=40 \
-device pcie-root-port,id=root2,slot=3,bus=bridge3 \
-monitor stdio \
-vnc :0 \


Thanks
Jing
Comment 7 errata-xmlrpc 2017-08-01 23:34:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 8 errata-xmlrpc 2017-08-02 01:12:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 9 errata-xmlrpc 2017-08-02 02:04:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 10 errata-xmlrpc 2017-08-02 02:45:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 11 errata-xmlrpc 2017-08-02 03:09:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 12 errata-xmlrpc 2017-08-02 03:29:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392