Bug 1238338

Summary: libvirt is overly restrictive with bus type and hotplug requirements on q35 QEMU machine
Product: Red Hat Enterprise Linux 7 Reporter: Laine Stump <laine>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.2CC: agedosier, alex.williamson, berrange, clalancette, dyuan, extras-qa, itamar, jforbes, laine, lhuang, libvirt-maint, mzhan, rbalakri, riddimshocker, shyu, veillard, virt-maint, yanyang
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-1.2.17-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1142382 Environment:
Last Closed: 2015-11-19 06:47:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1207751, 1340726    

Description Laine Stump 2015-07-01 15:54:59 UTC 
+++ This bug was initially created as a clone of Bug #1142382 +++

Description of problem:

For the majority of device types, libvirt assumes that the device is conventional PCI and requires hotplug, which currently makes is so they can only be attached behind a pci-to-pci bridge in a q35 machine.  The user should be able to override this and attach endpoint devices nearly wherever they please.  If the user specifies an address which is not hotplug capable, let them do so.  If they want to attach a conventional PCI device to a PCIe bus, do it.  libvirt should only be so restrictive when it is in charge of assigning devices to buses.


Version-Release number of selected component (if applicable):
1.2.8

How reproducible:
100%

Steps to Reproduce:
1. Attempt to assign a hostdev device to either the root complex (bus=0x0) or even directly behind the dmi bridge (bus=0x1)
2.
3.

Actual results:

libvirt enforces that a PCI device cannot be attached to a PCIe bus (even though my hostdev device very well might be PCIe) and requires the bus support hotplug (even though I've made no such requirement).

Expected results:

Let me do it

Additional info:

--- Additional comment from Laine Stump on 2014-09-16 13:58:36 EDT ---

Yep, this is the feedback I've been waiting for. I purposefully made the rules about plugging PCI devices into PCIe slots (and vice versa) and assigning to slots that aren't hot-pluggable restrictive to 1) avoid future problems in case qemu decided to start requiring that pci devices only be plugged into pci slots, and 2) avoid surprises when someon was unable to unplug a device.

I had *thought* that I already allowed manual assignment to a non-hotpluggable slot, but it sounds like not.

--- Additional comment from Jaroslav Reznik on 2015-03-03 12:03:58 EST ---

This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

--- Additional comment from Laine Stump on 2015-07-01 11:52:31 EDT ---

Patches have been pushed upstream (and will be in libvirt 1.2.17) to loosen the restrictions. libvirt will still attempt to auto-assign an address that is on a hotpluggable PCI bus, but youo can manually configure any address you want and libvirt will allow PCI<->PCIe as well as ignoring hotpluggability. (Unfortunately I forgot to reference this BZ in the commit log :-( ):

commit 1e15be1bbcbb0b43348aaabfb41b7a3a0fb51c43
Author: Laine Stump <laine>
Date:   Tue Jun 9 13:33:36 2015 -0400

    qemu: always permit PCI devices to be manually assigned to a PCIe bus

commit 9a12b6cd8c09df8328eac03c089a4f7302a71f11
Author: Laine Stump <laine>
Date:   Wed Jun 10 14:15:25 2015 -0400

    qemu: ignore assumptions about hotplug requirement when address is from config
Comment 3 Shanzhi Yu 2015-09-10 10:48:11 UTC 
Verify this bug with below steps

1. Allow PCI devices to be manually assigned to a PCIe bus

1.1 Prepare a guest with below xml 

# virsh dumpxml r7 
...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/r7.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/vdb.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/vdc.img'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x08' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
...

1.2 try to start guest 
# virsh start r7 
Domain r7 started

1.3 login guest to check guest pci device info 
# ssh root.122.231  lspci -t -v
root.122.231's password: 
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-02.0  Red Hat, Inc. QXL paravirtual graphic card
           +-03.0  Red Hat, Inc Virtio network device
           +-04.0  Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller
           +-06.0  Red Hat, Inc Virtio block device
           +-07.0  Red Hat, Inc Virtio memory balloon
           +-08.0  Red Hat, Inc Virtio block device
           +-1a.0-[01-02]--+-01.0-[02]----01.0  Red Hat, Inc Virtio console
           |               \-08.0  Red Hat, Inc Virtio block device
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

So vda and vdb were attached to PCIe bus, while vdc was attached to PCI bus

2. ignore assumptions about hotplug requirement when address is from config

2.1 Prepare a running guest with below xml 
# virsh dumpxml r7
..
  <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
..

2.2 Prepare three xml file as below
# cat vd{a,b,c}.xml
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/r7.img'/>
      <target dev='vda' bus='virtio'/>
    </disk>

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/vdb.img'/>
      <target dev='vdb' bus='virtio'/>
    </disk>

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/vdc.img'/>
      <target dev='vdc' bus='virtio'/>
    </disk>

2.3 Do hotplug
# virsh attach-device r7 vda.xml 
Device attached successfully

# virsh attach-device r7 vdb.xml 
Device attached successfully

# virsh attach-device r7 vdc.xml 
Device attached successfully

2.4 Check guest live xml

# virsh dumpxml r7
...
 <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/r7.img'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/vdb.img'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/vdc.img'/>
      <backingStore/>
      <target dev='vdc' bus='virtio'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
....

2.5 Check guest pci info 
# ssh root.122.231  lspci -t -v
root.122.231's password: 
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-02.0  Red Hat, Inc. QXL paravirtual graphic card
           +-03.0  Red Hat, Inc Virtio network device
           +-04.0  Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller
           +-07.0  Red Hat, Inc Virtio memory balloon
           +-1a.0-[01-02]----01.0-[02]--+-01.0  Red Hat, Inc Virtio console
           |                            +-02.0  Red Hat, Inc Virtio block device
           |                            +-03.0  Red Hat, Inc Virtio block device
           |                            \-04.0  Red Hat, Inc Virtio block device
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

Hi Laine,

Would you please help check if above steps is enough to verify this bug?

Thanks
Comment 4 Laine Stump 2015-09-11 16:20:59 UTC 
Yes, that is enough to verify that the fix is in the build.
Comment 5 Shanzhi Yu 2015-09-14 02:47:23 UTC 
Laine, Thanks

Verify this bug according comment 3 and comment 4.
Comment 6 Luyao Huang 2015-10-20 02:31:22 UTC 
Hi laine,

I found there are some problem during retest this bug:
"
   In short, after this patch, automatically PCI address assignment will
    assume that the device must be plugged in to a hot-pluggable slot, but
    manually assignment can place the device in any bus that is
    compatible, regardless of whether or not it supports hotplug. If the
    user makes a mistake and plugs the device into a bus that doesn't
    support hotplug, then later tries to do a hot-unplug, qemu will give
    an appropriate error.
"

From the commit message i think you want allow the user hot-plug a device to a bus even it does not support hotplug, and let qemu output error if qemu hot allow it. but the test result with latest libvirt totally different:

1.
# virsh dumpxml rhel7.0-rhel
...
  <os>
    <type arch='x86_64' machine='pc-q35-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
...
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
...


2. 

# cat disk.xml
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test6.img'/>
      <target dev='vdc' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1c' function='0x0'/>
    </disk>



# virsh attach-device rhel7.0-rhel disk.xml --live
error: Failed to attach device from disk.xml
error: internal error: PCI bus is not compatible with the device at 0000:01:1c.0. Device requires hot-plug capability, which is not provided by bus 0000:01

3.
# cat disk.xml
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test6.img'/>
      <target dev='vdc' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1c' function='0x0'/>
    </disk>

# virsh attach-device rhel7.0-rhel disk.xml --live
error: Failed to attach device from disk.xml
error: internal error: PCI bus is not compatible with the device at 0000:00:1c.0. Device requires a standard PCI slot, which is not provided by bus 0000:00


Would you please help to check if the test result is except ? if not, can we set FailQA to this bug ? Thanks in advance for your reply.

Thanks,
Luyao
Comment 7 Luyao Huang 2015-10-21 02:22:50 UTC 
Sorry for the noise, i misunderstood the design, you just want to fix the config and cold-plug part not include the hot-plug/hot-unplug case.
Comment 9 errata-xmlrpc 2015-11-19 06:47:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2202.html