It happens as well when hot-plugging devices into root-port

Steps like this
1. start a vm with 1 root-port
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1f' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='3' port='0x10'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>

2. hot-plug 1 nic into root-port

# cat nic.xml 
<interface type='network'>
      <source network='default' bridge='virbr0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='3' slot='0' function='0x0'/>
    </interface>

# virsh attach-device vm1-q35 nic.xml 
error: Failed to attach device from nic.xml
error: internal error: PCI bus is not compatible with the device at 0000:03:00.0. Device requires a standard PCI slot, which is not provided by bus 0000:03

I see the problem:

1) All of the hotplug functions set flags to indicate the device needs a bus with legacy ("standard") PCI slots. Then they call virDomainPCIAddressEnsureAddr()


2) virDomainPCIAddressEnsureAddr() checks for a "compatible" bus type, but sets the "address is from Config" flag, which has the effect of accepting either PCI or PCIe buses. (i.e. so far we're doing okay). Then it calls...

3) virDomainPCIAddressReserveSlot()l but that function doesn't have an argument for "address is from Config", it just knows we want a "legacy PCI slot". It then calls:

4) virDomainPCIAddressReserveAddr(), which *does* have a "address is from Config" argument, but since virDomainPCIAddressReserveSlot() doesn't know what the callers wanted, it assumes "address is internally auto-generated by libvirt". Because of this, our request for "legacy PCI" is interpreted very strictly, and the pcie-root-port fails the validation.

All of this code will be changing soon, but a simple fix is possible.

The fix is a single line, and has no potential for causing new problems. I will have the patch posted/reviewed/pushed upstream and backported within the next day (I just want to test it first :-)

The lack of this fix makes it impossible to hotplug devices into a PCI Express port on Q35. Since hotplug into legacy PCI slots also doesn't work on Q35, not having the fix means that hotplug of PCI devices won't work at all on Q35, so I'm proposing as a blocker. If this is more appropriate as a 0-day or something that's fine too.

Fix pushed upstream:

commit b87703cf79559157404667628802d7fe8f9f19a6
Author: Laine Stump <laine>
Date:   Fri Sep 9 15:26:34 2016 -0400

    conf: allow hotplugging "legacy PCI" device to manually addressed PCIe slot

Verified with libvirt-2.0.0-9.el7.x86_64 and qemu-kvm-rhev-2.6.0-25.el7.x86_64

1. Hotplug/hot-unplug virtio disk into/from root-port
#virsh dumpxml vm1-q35
<controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1f' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='3' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>

# virsh attach-disk vm1-q35 /var/lib/libvirt/images/test.img vdb --subdriver qcow2 --address pci:0.3.0.0
Disk attached successfully

# virsh dumpxml vm1-q35 | grep vdb -a6
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>

check the disk in guest, it is usable
# mkfs.xfs /dev/vdb -f
meta-data=/dev/vdb               isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
# mount /dev/vdb /mnt
# echo hello > /mnt/hello
# cat /mnt/hello 
hello
# umount /mnt

# virsh detach-disk vm1-q35 vdb
Disk detached successfully

check domain xml, vdb vanishes

2. Hot-plug/Hot-unplug nic into/from root-port
# virsh attach-device vm1-q35 nic.xml 
Device attached successfully

# cat nic.xml 
<interface type='network'>
      <source network='default' bridge='virbr0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='3' slot='0' function='0x0'/>
    </interface>

# virsh dumpxml vm1-q35 | grep inter
<interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>

check in guest
# ifconfig
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet x.x.x.x  netmask 255.255.255.0  broadcast x.x.x.x
        inet6 fe80::5054:ff:fe3a:819a  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:3a:81:9a  txqueuelen 1000  (Ethernet)
        RX packets 9  bytes 1670 (1.6 KiB)
        RX errors 296  dropped 0  overruns 0  frame 296
        TX packets 10  bytes 2378 (2.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# virsh detach-device vm1-q35 nic.xml 
Device detached successfully

check in domain xml, nic vanishes

3. Hot-plug/Hot-unplug virtio disk into/from downstream port
# virsh dumpxml vm1-q35
<controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1f' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='3' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-switch-upstream-port'>
      <model name='x3130-upstream'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-switch-downstream-port'>
      <model name='xio3130-downstream'/>
      <target chassis='5' port='0x0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>

# virsh attach-disk vm1-q35 /var/lib/libvirt/images/test.img vdb --subdriver qcow2 --address pci:0.5.0.0
Disk attached successfully

# virsh dumpxml vm1-q35 | grep vdb -a6
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test.img'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>

***The disk cannot be detected in guest. Checked with qemu qe, she thought it's caused by BZ1365613***

# virsh detach-disk vm1-q35 vdb
Disk detached successfully

check domain xml, vdb vanishes

4. Hot-plug/Hot-unplug nic into/from downstream port
# cat nic.xml 
<interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </interface>

# virsh attach-device vm1-q35 nic.xml 
Device attached successfully

# ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet x.x.x.x  netmask 255.255.255.0  broadcast x.x.x.x
        inet6 fe80::5054:ff:fe3a:819a  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:3a:81:9a  txqueuelen 1000  (Ethernet)
        RX packets 22  bytes 3443 (3.3 KiB)
        RX errors 7  dropped 0  overruns 0  frame 7
        TX packets 40  bytes 4948 (4.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# virsh detach-device vm1-q35 nic.xml 
Device detached successfully

check domain xml, nic vanishes

Got the similar test results under OVMF.

Laine,

I attempted to hot-plug nic into pcie-root-port which is attached into pcie-expander-bus (e.g. pcie-pxb --> pcie-root-port). But I cannot detect the nic in guest. Then I hot-unplugged the nic. Virsh commands returns w/o error but the nic is not unplugged from domain xml. Virtio disk has the similar problem.

I attempted to hot-plug nic into downstream-port which is attached into pcie-expander-bus (e.g. pcie-pxb --> pcie-root-port --> upstream-port --> downstream port). I cannot detect the nic in guest as well. After hot-unplugging nic, the nic xml still exists in domain xml. Virtio disk has the similar problem.

Steps are as follows:
1. start vm with following xml
<controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='100'>
        <node>1</node>
      </target>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='4' port='0x0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>

2. hot-plug nic into root-port
# cat nic.xml 
<interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

# virsh attach-device vm1-q35-Sec nic.xml 
Device attached successfully

check domain xml, there are 2 nics
# virsh dumpxml vm1-q35-Sec | grep inter -a6
      
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
   
checked in guest, only 1 nic is found. I cannot detect the hot-plugged nic
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.4.171  netmask 255.255.252.0  broadcast 10.66.7.255
        inet6 fe80::5054:ff:fe65:18d8  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:65:18:d8  txqueuelen 1000  (Ethernet)
        RX packets 133  bytes 10172 (9.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15  bytes 1986 (1.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

3. hot-unplug the nic
# virsh detach-device vm1-q35-Sec nic.xml 
Device detached successfully

4. checked in domain xml. The nic xml still exists in domain xml

# virsh dumpxml vm1-q35-Sec | grep inter -a6
     
    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:3a:81:9a'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

libvirt is allowing the hotplug into the given port, and passing the command through to qemu. So if the guest isn't recognizing it, it's a problem with the combination of pcie-expander-bus + pcie-root-port + hotplug, either in the guest OS or in qemu.

(BTW, it's normal for the device to still be listed in the libvirt XML after an unplug in a situation like this - because the guest never saw the device in the first place, it hasn't done [whatever it is that guests do to respond to a hotplug request], so qemu hasn't been notified by the guest that it's finished unplugging the device, and thus qemu hasn't notified libvirt with a DEVICE_DELETED event, so libvirt still shows the device as in-use by the guest)

Marcel, do you have any advice or ideas?

(in the meantime, this isn't a separate problem from what this BZ was filed for, so you should open a new BZ saying that when you hotplug to a pcie-root-port connected to a pcie-expander-bus, the hotplug is successful but the guest doesn't see the new device).

Thanks Laine.

Opened a qemu bz1377160 to track the problem of pcie-pxb+pcie-root-port+hot-plug

According to comment#7, move it to verified status

Marcel answered my question in Bug 1377160, so I'm clearing needinfo.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html