+++ This bug was initially created as a clone of Bug #1408808 +++ +++ This bug was initially created as a clone of Bug #1390316 +++ Two reasons: 1. Can be used by both x86 and ARM VMs 2. Add a vendor-specific pci capability to these ports, with recommended window sizes (io/mem/prefmem) for the firmware for more efficient resource allocation. Add libvirt support for the new controller.
The new controller will be selected automatically by libvirt for new setups, but nova should be aware the Intel Root Port is being replaced by a generic one. Not sure if there are any actual work items for this BZ.
There's no clear explanation as to why this feature is required or what use cases it will resolve. Until such a time as this is provided, I'm going to mark this as closed.
(In reply to Stephen Finucane from comment #4) > There's no clear explanation as to why this feature is required or what use > cases it will resolve. From parsing comment 1 Requirement: track libvirt changes to ensure nova-libvirt integration does not introduce bugs - fix nova when necessary Impacted use cases: all "new setups" > Until such a time as this is provided, I'm going to > mark this as closed. IMHO, a request for more information should always be made before closing a bug as insufficient information. I don't see any request. Was one made outside the visibility of this BZ? Note, if you look at bug 1410437, you'll see how ovirt approached their clone of the same bug. They marked it as test-only and moved it to ON_QA. After ensuring there were no problems with the libvirt change, the bug was eventually closed with the more satisfactory ERRATA resolution. Thanks, drew
Re-opening this bug as per Andrew Jones' comment#5
Hi Andrea, I somehow have an inkling that you might be able to tell more about what to keep in mind when testing this, from a libvirt point of view.
(In reply to Kashyap Chamarthy from comment #7) > Hi Andrea, > > I somehow have an inkling that you might be able to tell more about what to > keep in mind when testing this, from a libvirt point of view. libvirt has switched to the new controller more than a year ago, so unless Nova is doing something very silly like explicitly specifying the concrete model like <controller type='pci' model='pcie-root-port'> <model name='ioh3420'/> </controller> you're almost certainly going to be fine.
(In reply to Andrea Bolognani from comment #8) > (In reply to Kashyap Chamarthy from comment #7) > > Hi Andrea, > > > > I somehow have an inkling that you might be able to tell more about what to > > keep in mind when testing this, from a libvirt point of view. > > libvirt has switched to the new controller more than a year ago, > so unless Nova is doing something very silly like explicitly > specifying the concrete model like > > <controller type='pci' model='pcie-root-port'> > <model name='ioh3420'/> > </controller> > > you're almost certainly going to be fine. Thanks, Andrea! Yep, Nova doesn't configure an explicit PCIe controller model name like that. All Nova currently allows is to permit configuring PCIe root ports. From nova/conf/libvirt.py: ----------------------------------------------------------------------- [...] cfg.IntOpt('num_pcie_ports', default=0, min=0, max=28, help= """ The number of PCIe ports an instance will get. Libvirt allows a custom number of PCIe ports (pcie-root-port controllers) a target instance will get. Some will be used by default, rest will be available for hotplug use. By default we have just 1-2 free ports which limits hotplug. More info: https://github.com/qemu/qemu/blob/master/docs/pcie.txt Due to QEMU limitations for aarch64/virt maximum value is set to '28'. Default value '0' moves calculating amount of ports to libvirt. [...] -----------------------------------------------------------------------
(In reply to Kashyap Chamarthy from comment #9) > > libvirt has switched to the new controller more than a year ago, > > so unless Nova is doing something very silly like explicitly > > specifying the concrete model like > > > > <controller type='pci' model='pcie-root-port'> > > <model name='ioh3420'/> > > </controller> > > > > you're almost certainly going to be fine. > > Thanks, Andrea! Yep, Nova doesn't configure an explicit PCIe controller > model name like that. > > All Nova currently allows is to permit configuring PCIe root ports. Right. I just remembered that I provided suggestions and feedback at the time the support was being added to Nova with https://review.openstack.org/#/c/545034/ so it makes sense that the current implementation would be fairly reasonable O:-)
[Thanks to Stephen Finucane for doing a couple of tests for me on DevStack] Joe, to test this: (1) Ensure you boot the guest with Q35 machine type. This can be done in two ways: (a) by setting `hw_machine_type=x86_64=q35` in Nova config under the [libvirt] section; or (b) by setting the metadata property on the Glance image, with which you will boot the guest: `openstack image set --property hw_machine_type=x86_64=q35 Fedora28Template`: $ openstack flavor create test.q35 $ openstack flavor set --property hw:machine_type=x86_64=q35 test.q35 (2) Then, set `num_pcie_ports=2` under [libvirt] section in nova.conf. (3) And boot a guest with the flavor created in step (1): # Set the apprpriate NIC UUID $ openstack server create --flavor test.q35 --image test \ --nic net-id=$NIC_UUID test-q35 (4) Once the guest is up and running, check `virsh dumpxml test-q35`, you should see something like the following for the PCI controllers: [...] <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> [...] <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller>
(In reply to Kashyap Chamarthy from comment #12) > [Thanks to Stephen Finucane for doing a couple of tests for me on DevStack] > > Joe, to test this: > > (1) Ensure you boot the guest with Q35 machine type. This can be done > in two ways: (a) by setting `hw_machine_type=x86_64=q35` in Nova > config under the [libvirt] section; or (b) by setting the metadata > property on the Glance image, with which you will boot the guest: > `openstack image set --property hw_machine_type=x86_64=q35 > Fedora28Template`: > > $ openstack flavor create test.q35 > $ openstack flavor set --property hw:machine_type=x86_64=q35 test.q35 > > (2) Then, set `num_pcie_ports=2` under [libvirt] section in nova.conf. Not knowing anything about Nova, the test methodology in general looks pretty good, but setting num_pcie_ports to 2 seems a bit pointless: a very basic q35 guest will probably use 4 PCIe Root Ports for disk, network interface, memory balloon and RNG device respectively; in fact, as you can see... > (3) And boot a guest with the flavor created in step (1): > > # Set the apprpriate NIC UUID > $ openstack server create --flavor test.q35 --image test \ > --nic net-id=$NIC_UUID test-q35 > > (4) Once the guest is up and running, check `virsh dumpxml test-q35`, > you should see something like the following for the PCI controllers: > > [...] > <controller type='pci' index='0' model='pcie-root'> > <alias name='pcie.0'/> > </controller> > <controller type='pci' index='1' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='1' port='0x10'/> > <alias name='pci.1'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x0' multifunction='on'/> > </controller> > <controller type='pci' index='2' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='2' port='0x11'/> > <alias name='pci.2'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x1'/> > </controller> > [...] > <controller type='pci' index='3' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='3' port='0x12'/> > <alias name='pci.3'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x2'/> > </controller> > <controller type='pci' index='4' model='pcie-root-port'> > <model name='pcie-root-port'/> > <target chassis='4' port='0x13'/> > <alias name='pci.4'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > function='0x3'/> > </controller> ... you already have (at least) 4 of them :) So the setting above is basically ignored because libvirt will automatically allocate more PCIe Root Ports as needed: a better test would be to set it to a higher value like 8 and verify that once the guest has booted there are a few spare ones that can be used for hotplugging additional devices. As an aside, if I'm reading the above correctly, it seems like you can define a "flavor" that will use q35, but you can only pick the number of PCIe Root Ports that should be added to such guests through a global knob: I would expect the number of PCIe Root Ports to also be configured in the flavor. Does that not work?
(In reply to Andrea Bolognani from comment #13) > (In reply to Kashyap Chamarthy from comment #12) [...] > > (1) Ensure you boot the guest with Q35 machine type. This can be done > > in two ways: (a) by setting `hw_machine_type=x86_64=q35` in Nova > > config under the [libvirt] section; or (b) by setting the metadata > > property on the Glance image, with which you will boot the guest: > > `openstack image set --property hw_machine_type=x86_64=q35 > > Fedora28Template`: > > > > $ openstack flavor create test.q35 > > $ openstack flavor set --property hw:machine_type=x86_64=q35 test.q35 Correction: the above two commands are wrong, as setting machine type via flavor "Extra Specs" (key and value pairs that define on which compute nodes a flavor can run) is a no-op. The correct way is via either of the two methods noted in the description. I.e. you can set machine type in either of the two ways: (a) By setting it in the /etc/nova.conf on the Compute node: ... [libvirt] hw_machine_type=x86_64=q35 ... (b) By setting the metadata property on the "Glance" template image, with which you will boot a guest: $ openstack image set \ --property hw_machine_type=x86_64=q35 Fedora28Template` > > (2) Then, set `num_pcie_ports=2` under [libvirt] section in nova.conf. > > Not knowing anything about Nova, the test methodology in general > looks pretty good, but setting num_pcie_ports to 2 seems a bit > pointless: a very basic q35 guest will probably use 4 PCIe Root > Ports for disk, network interface, memory balloon and RNG device > respectively; in fact, as you can see... Right, I actually first wrote: "optionally set the `num_pcie_ports`, for the reasons you state above. I've observed it in a test guest XML with plain Q35. So indeed step-2 can be elided — unless setting it to a higher value as '8', as you note further below. [...] > > (4) Once the guest is up and running, check `virsh dumpxml test-q35`, > > you should see something like the following for the PCI controllers: > > [...] > > <controller type='pci' index='4' model='pcie-root-port'> > > <model name='pcie-root-port'/> > > <target chassis='4' port='0x13'/> > > <alias name='pci.4'/> > > <address type='pci' domain='0x0000' bus='0x00' slot='0x02' > > function='0x3'/> > > </controller> > > ... you already have (at least) 4 of them :) > > So the setting above is basically ignored because libvirt will > automatically allocate more PCIe Root Ports as needed: a better > test would be to set it to a higher value like 8 and verify > that once the guest has booted there are a few spare ones that > can be used for hotplugging additional devices. Yeah, sounds good. :-) Thanks for the review, Andrea! (Just to note: the above XML is still correct, and generated by setting the machine type via the global config knob.) > As an aside, if I'm reading the above correctly, it seems like > you can define a "flavor" that will use q35, That "flavor" configuration is a no-op currently. Sorry for the mix-up. The only two valid ways to configure a machine type are the ones that I noted earlier (global config knob on each Compute node; or a property set on per disk image template, which is used to boot a Nova "instance"). > but you can only pick the number of PCIe Root Ports that should be > added to such guests through a global knob: I would expect the number > of PCIe Root Ports to also be configured in the flavor. Does that not > work? No, not at present. So, to summarize the current status: - Machine Type: Can be configured via: (a) a per-Compute node config attribute in nova.conf; and (b) via the Glance template image metadata property. - PCIe Root Ports: Can be configured only via the per-Compute. The potential TODO item here would be: * Allow configuring PCIe Root Ports via the Glance template image metadata property. * Optionally, allow configuring PCIe Root Ports _and_ machine type via the flavor "extra specs" (https://docs.openstack.org/nova/latest/user/flavors.html#extra-specs)
Joe, so here are the updated test instructions: (1) Ensure you boot the guest with Q35 machine type. This can be done in two ways: (a) By setting it in the `/etc/nova.conf` on the Compute node: ... [libvirt] hw_machine_type=x86_64=q35 ... (b) By setting the metadata property on the "Glance" template image, with which you will boot a guest: $ openstack image set \ --property hw_machine_type=q35 Fedora28Template` (2) Optionally, set `num_pcie_ports=8` under [libvirt] section in nova.conf. (3) Boot an instance. NB: If you've set the machine type in `/etc/nova.conf` on a Compute node, then booting an instance with any template is fine; but if you set the machine type on a particular Glance image, use that image to boot the Nova instance: # Set the apprpriate NIC UUID $ openstack server create --flavor 2 --image test \ --nic net-id=$NIC_UUID test-q35 (4) Once the guest is up and running, run `virsh dumpxml test-q35`, you should see a 'q35' machine type (depends on the QEMU version you're running), about 8 PCIe root ports (I'm only showing 4 below for bevity's sake): [...] <os> <type arch='x86_64' machine='pc-q35-2.11'>hvm</type> <boot dev='hd'/> <smbios mode='sysinfo'/> </os> [...] <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> [...] <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x16'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='8' port='0x17'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller> [...]
(In reply to Kashyap Chamarthy from comment #14) > > As an aside, if I'm reading the above correctly, it seems like > > you can define a "flavor" that will use q35, > > That "flavor" configuration is a no-op currently. Sorry for the mix-up. > The only two valid ways to configure a machine type are the ones that I > noted earlier (global config knob on each Compute node; or a property > set on per disk image template, which is used to boot a Nova > "instance"). > > > but you can only pick the number of PCIe Root Ports that should be > > added to such guests through a global knob: I would expect the number > > of PCIe Root Ports to also be configured in the flavor. Does that not > > work? > > No, not at present. > > So, to summarize the current status: > > - Machine Type: Can be configured via: (a) a per-Compute node config > attribute in nova.conf; and (b) via the Glance template image > metadata property. > > - PCIe Root Ports: Can be configured only via the per-Compute. > > The potential TODO item here would be: > > * Allow configuring PCIe Root Ports via the Glance template image > metadata property. > > * Optionally, allow configuring PCIe Root Ports _and_ machine type > via the flavor "extra specs" > (https://docs.openstack.org/nova/latest/user/flavors.html#extra-specs) Assuming I haven't gotten the basic idea behind flavors wrong, the latter seems especially interesting; as for the former, I can't imagine it being too relevant for the time being, as most (all?) guest images you could reasonably want to use will boot just fine on both i440fx and q35 virtual hardware and won't require any particular number of PCIe Root Ports to be present in the second case.
(In reply to Kashyap Chamarthy from comment #15) [...] > (2) Optionally, set `num_pcie_ports=8` under [libvirt] section in > nova.conf. [...] > (4) Once the guest is up and running, run `virsh dumpxml test-q35`, > you should see a 'q35' machine type (depends on the QEMU version you're > running), about 8 PCIe root ports More precisely, you should see *at least* 'num_pcie_ports' PCIe Root Ports if you have set the corresponding value in step (2) or an arbitrary, probably quite small, number otherwise.
closing as not a bug as there is nothing that we reasonably should do to adress this in nova/openstack. nova never asserted or specified the model of the root port so we really dont care that this has changed unles it would break customers. if it would the 1 it would be a libvirt regression and 2 we would have to adress it in nova via a workaround. that is not the case so closing this out.