RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1377160 - [RFE] Q35: Implement hotplug for pxb-pcie devices
Summary: [RFE] Q35: Implement hotplug for pxb-pcie devices
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Marcel Apfelbaum
QA Contact: jingzhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-19 02:18 UTC by Yang Yang
Modified: 2017-08-02 03:29 UTC (History)
9 users (show)

Fixed In Version: qemu-kvm-rhev-2.9.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 23:34:44 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1361059 0 medium CLOSED [RFE] Implement hotplug for pxb devices 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHSA-2017:2392 0 normal SHIPPED_LIVE Important: qemu-kvm-rhev security, bug fix, and enhancement update 2017-08-01 20:04:36 UTC

Internal Links: 1361059

Description Yang Yang 2016-09-19 02:18:08 UTC 
Description of problem:
When hotplugging to a pcie-root-port connected to a pcie-expander-bus (i.e. pcie-pxb --> pcie-root-port), the hotplug is successful but the guest doesn't see the new device. 
See the similar problem when hot-plugging to pcie-switch-downstream-port connected to pcie-expander-bus (i.e. pcie-pxb --> pcie-root-port --> pcie-switch-upstream-port --> pcie-switch-downstream-port)

Version-Release number of selected component (if applicable):
libvirt-2.0.0-9.el7.x86_64
qemu-kvm-rhev-2.6.0-25.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. start vm with pcie-pxb --> pcie-root-port
<controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='100'>
        <node>1</node>
      </target>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='4' port='0x0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>

2. hot-plug nic into root-port
# cat nic.xml 
<interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

# virsh attach-device vm1-q35-Sec nic.xml 
Device attached successfully

check domain xml, there are 2 nics
# virsh dumpxml vm1-q35-Sec | grep inter -a6
      
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
   
checked in guest, only 1 nic is found. Guest cannot see the hot-plugged nic

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.4.171  netmask 255.255.252.0  broadcast 10.66.7.255
        inet6 fe80::5054:ff:fe65:18d8  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:65:18:d8  txqueuelen 1000  (Ethernet)
        RX packets 133  bytes 10172 (9.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15  bytes 1986 (1.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

3. hot-unplug the nic
# virsh detach-device vm1-q35-Sec nic.xml 
Device detached successfully

4. checked in domain xml. The nic xml still exists in domain xml

# virsh dumpxml vm1-q35-Sec | grep inter -a6
     
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

Actual results:
When hot-plugging to root-port(or downstream-port) connected to pcie-pxb, hotplug is successful but guest cannot see the new device

Expected results:
When hot-plugging to root-port(or downstream-port) connected to pcie-pxb, hotplug is successful and guest can see the new device

Additional info:
Comment 3 Karen Noel 2016-09-21 00:46:11 UTC 
This seems like something that should work for Q35. Is it a bug or missing functionality?
Comment 4 Marcel Apfelbaum 2016-09-28 09:58:51 UTC 
(In reply to Karen Noel from comment #3)
> This seems like something that should work for Q35. Is it a bug or missing
> functionality?

Hot-plugging/unplugging for devices attached to Root Ports/Downstream ports
of the main host bridge (Root Complex) works OK.

The hot-plug support for extra host bridges (pxb/pxb-pcie) is not implemented yet. 


Thanks,
Marcel
Comment 5 jingzhao 2017-04-24 09:10:50 UTC 
Reproduce the bz on qemu-kvm-rhev-2.8.0-6.el7.x86_64

Verified the bz on qemu-kvm-rhev-2.9.0-1.el7.x86_64

Following are the detailed info

1. Boot guest with qemu command line [1]

2. Hot-plug nic device with qmp command:
{"execute": "netdev_add", "arguments": { "type":"tap","id":"net1"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"virtio-net-pci","netdev":"net1","mac":"22:11:22:45:61:98","id":"net1","bus":"root2"}}
{"return": {}}


3. In guest, check the nic device works well
[root@localhost ~]# lspci 
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04)
00:02.0 Host bridge: Red Hat, Inc. Device 000b
00:03.0 Host bridge: Red Hat, Inc. Device 000b
00:04.0 Host bridge: Red Hat, Inc. Device 000b
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
08:00.0 PCI bridge: Red Hat, Inc. Device 000c
09:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
14:00.0 PCI bridge: Red Hat, Inc. Device 000c
15:00.0 SCSI storage controller: Red Hat, Inc Virtio block device (rev 01)
28:00.0 PCI bridge: Red Hat, Inc. Device 000c
29:00.0 Ethernet controller: Red Hat, Inc Virtio network device (rev 01)
[root@localhost ~]# lspci -vvv -t
-+-[0000:28]---00.0-[29]----00.0  Red Hat, Inc Virtio network device
 +-[0000:14]---00.0-[15]----00.0  Red Hat, Inc Virtio block device
 +-[0000:08]---00.0-[09]----00.0  Red Hat, Inc Virtio block device
 \-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
             +-01.0  Red Hat, Inc. QXL paravirtual graphic card
             +-02.0  Red Hat, Inc. Device 000b
             +-03.0  Red Hat, Inc. Device 000b
             +-04.0  Red Hat, Inc. Device 000b
             +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
             +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
             \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:09\:00.0/numa_node 
0
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:15\:00.0/numa_node 
1
[root@localhost ~]# cat /sys/bus/pci/devices/0000\:29\:00.0/numa_node 
2
[root@localhost ~]# ping 10.66.4.211
PING 10.66.4.211 (10.66.4.211) 56(84) bytes of data.
64 bytes from 10.66.4.211: icmp_seq=1 ttl=64 time=1.76 ms
64 bytes from 10.66.4.211: icmp_seq=2 ttl=64 time=0.925 ms
^C
--- 10.66.4.211 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.925/1.345/1.765/0.420 ms

4. Unplug the nic device in hmp
(qemu) netdev_del net1   
(qemu) device_del net1 

5. Chekc the nic in guest, and nic device can be deleted

[1]
/usr/libexec/qemu-kvm \
-machine q35,smm=on,accel=kvm \
-cpu Haswell-noTSX \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-object memory-backend-ram,size=1024M,id=ram-node0 \
-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
-object memory-backend-ram,size=1024M,id=ram-node1 \
-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
-object memory-backend-ram,size=2048M,id=ram-node2 \
-numa node,nodeid=2,cpus=2,memdev=ram-node2 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-serial unix:/tmp/serial0,server,nowait \
-boot menu=on \
-qmp tcp:0:6666,server,nowait \
-vga qxl \
-chardev file,path=/home/seabios.log,id=seabios -device isa-debugcon,chardev=seabios,iobase=0x402 \
-device pxb-pcie,id=bridge1,bus=pcie.0,numa_node=0,bus_nr=8 \
-device pcie-root-port,bus=bridge1,id=root0,slot=1 \
-drive file=/home/test/rhel/rhel74.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=root0,bootindex=0 \
-device pxb-pcie,id=bridge2,bus=pcie.0,numa_node=1,bus_nr=20 \
-device pcie-root-port,bus=bridge2,id=root1,slot=2 \
-drive file=/home/test/rhel/block1.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk1,id=virtio-disk1,bus=root1 \
-device pxb-pcie,id=bridge3,bus=pcie.0,numa_node=2,bus_nr=40 \
-device pcie-root-port,id=root2,slot=3,bus=bridge3 \
-monitor stdio \
-vnc :0 \


Thanks
Jing
Comment 7 errata-xmlrpc 2017-08-01 23:34:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 8 errata-xmlrpc 2017-08-02 01:12:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 9 errata-xmlrpc 2017-08-02 02:04:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 10 errata-xmlrpc 2017-08-02 02:45:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 11 errata-xmlrpc 2017-08-02 03:09:50 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Comment 12 errata-xmlrpc 2017-08-02 03:29:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2392
Note You need to log in before you can comment on or make changes to this bug.