Created attachment 1713193 [details]
Bridge_before_after_reboot

Created attachment 1713194 [details]
sriov_before_after_reboot

Created attachment 1714382 [details]
Host side domxml and lspci

Created attachment 1714383 [details]
Guest side lspci and dmesg

Comparing the domxml with the actual VM PCI devices, there seem to be a problem with the PCI addresses.
They are not in sync for some reason, even on first boot when everything seems fine (e.g. the interface is there).

Attaching the relevant information from the host (virt launcher `virsh dumpxml 1`) and the guest (From the VM, `lspci`).

Just a short recap from discussions with Edward on IRC:

1) The observed mismatch in bus numbers between the libvirt XML and guest lspci output is because the "bus number" in libvirt is valid/consistent only within the libvirt config. qemu itself doesn't provide a way to explicitly specify a bus number for any given PCI controller, because it has no way of enforcing such a request - the bus number of any given PCI controller as reported by any guest OS is completely up to the guest OS; it usually depends on the order in which the controllers are probed, but that's not anything that's enforced, and anyway the guest OS could probe controllers in any order it wanted to.

2) Note that the two devices do have the same PCI address from one boot to the next, and the systemd-provided "predictable"  device names remain consistent (as shown by "altname" in the ifconfig output): The device with MAC 02:00:b5:b5:b5:fb is "enp2s1" both times, and the device with MAC 02:57:92:00:00:04 is "enp3s0" both times.

3) But for some reason, this guest has been configured to use traditional ethN network interface names instead of systemd-provided "predictable network interface names". Use of ethN device names has proven problematic (i.e. "unpredictable and prone to change order from one boot to the next on identical hardware") over many years, which is why systemd provides the "predictable" names. Here is a document that explains the problem, and why systemd came to the conclusion that it's not possible to have predictable names using the ethN naming scheme (unless you only have a single ethernet device):

  https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

IMO, the proper solution to this is to *not* use ethN network interface names when there is more than a single network device.

(In reply to Laine Stump from comment #10)
> Just a short recap from discussions with Edward on IRC:
> 
> 1) The observed mismatch in bus numbers between the libvirt XML and guest
> lspci output is because the "bus number" in libvirt is valid/consistent only
> within the libvirt config. qemu itself doesn't provide a way to explicitly
> specify a bus number for any given PCI controller, because it has no way of
> enforcing such a request - the bus number of any given PCI controller as
> reported by any guest OS is completely up to the guest OS; it usually
> depends on the order in which the controllers are probed, but that's not
> anything that's enforced, and anyway the guest OS could probe controllers in
> any order it wanted to.

That's a bit surprising to me.

Daniel, doesn't this statement invalidates the assumption in Device role tagging?
[1] https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-device-role-tagging.html


> 
> 2) Note that the two devices do have the same PCI address from one boot to
> the next, and the systemd-provided "predictable"  device names remain
> consistent (as shown by "altname" in the ifconfig output): The device with
> MAC 02:00:b5:b5:b5:fb is "enp2s1" both times, and the device with MAC
> 02:57:92:00:00:04 is "enp3s0" both times.
> 
> 3) But for some reason, this guest has been configured to use traditional
> ethN network interface names instead of systemd-provided "predictable
> network interface names". Use of ethN device names has proven problematic
> (i.e. "unpredictable and prone to change order from one boot to the next on
> identical hardware") over many years, which is why systemd provides the
> "predictable" names. Here is a document that explains the problem, and why
> systemd came to the conclusion that it's not possible to have predictable
> names using the ethN naming scheme (unless you only have a single ethernet
> device):
> 
>  
> https://www.freedesktop.org/wiki/Software/systemd/
> PredictableNetworkInterfaceNames/
> 
> IMO, the proper solution to this is to *not* use ethN network interface
> names when there is more than a single network device.

(In reply to Vladik Romanovsky from comment #11)
> (In reply to Laine Stump from comment #10)
> > Just a short recap from discussions with Edward on IRC:
> > 
> > 1) The observed mismatch in bus numbers between the libvirt XML and guest
> > lspci output is because the "bus number" in libvirt is valid/consistent only
> > within the libvirt config. qemu itself doesn't provide a way to explicitly
> > specify a bus number for any given PCI controller, because it has no way of
> > enforcing such a request - the bus number of any given PCI controller as
> > reported by any guest OS is completely up to the guest OS; it usually
> > depends on the order in which the controllers are probed, but that's not
> > anything that's enforced, and anyway the guest OS could probe controllers in
> > any order it wanted to.
> 
> That's a bit surprising to me.
> 
> Daniel, doesn't this statement invalidates the assumption in Device role
> tagging?
> [1]
> https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-
> device-role-tagging.html

Yes, it is a bit more complex than described in that spec. The bus numbers in libvirt XML and guest are not a 1-1 match as originally supposed. In simple PCI topologies we can make some simplifying assumptions and it'll still work. For more complex toplogies we need extra information to reliably correlate.

There is actually a separate "bus_nr" setting in QEMU against some of the their PCI bridges that is in turn exposed to the guest OS. If libvirt supported setting that field, then the host/guest correlation can be made more reliable.

> There is actually a separate "bus_nr" setting in QEMU against some of the
> their PCI bridges that is in turn exposed to the guest OS. If libvirt
> supported setting that field, then the host/guest correlation can be made
> more reliable.

The only PCI controllers in qemu that have a "bus_nr" setting are pci-expander-bus and pcie-expander-bus, and libvirt already supports setting it (via <target BusNr='n'/> in the controller XML). But that isn't the purpose of bus_nr for those controllers. bus_nr is set to split the bus numbering space (which is an 8 bit value) between the pcie-root-bus and its descendants, and the pcie-expander-bus and *its* descendants. So for example, if you set <target busNr='100'/> in the definition of the pcie-expander-bus, then bus numbers 1-99 will be available for descendants of the root bus, and bus numbers 100-255 will be available for descendants of the pcie-expander-bus.

There are settings called "chassis" and/or "chassis_nr" for some PCI controllers (and libvirt also supports setting those in the <target> subelement, for controllers that have them), but those also have nothing to do with the bus number. Setting those apparently will set a register in the PCI controller, and that register can be seen from within the guest OS, but that register has no effect on the bus number reported by the guest OS. (I was unsure of this so I just tried setting chassis on the pcie-root-ports of a guest and restarting it - the output of lspci was unchanged)

So unfortunately I don't think there is a way for qemu to directly/explicitly set the bus number of any PCI controller.

The reported issue has been investigated and finally classified as a NOTABUG.
The issue with the lost NIC interface configuration and its inconsistent renaming due to reordering is due to the usage of the non-predictable interface naming.

Summarizing the end result of this issue investigation:
- The initial identification of the PCI address inconsistency between the domxml and the VM actual mapping:
  - Although in most case the mapping is in sync (between libvirt & the VM) , in some cases there is no way to assure it. Therefore, it is not to be expected for the mapping sync to be assured.
  - In the current case, the inconsistency addresses are caused by the wrong identification of the VF as a PCI device and not a PCIe device (see next for details).
  - The PCI mapping sync is not expected to cause issues with the VF functionality or performance.
- libvirt identifies the VF device as PCI instead of PCIe:
  This is a libvirt issue that is caused by its inability to access the VF device information in the unprivileged container.
  libvirt works in a privileged mode in the virt-launcher, but the container itself is not. For libvirt to work properly in such a case, it should have been launched in an unprivileged/user mode or not assume full access in a privileged mode.
  In any case, this is a side effect issue found, it should not really affect the real problem reported in this BZ (see previous details).
- There is no mechanism which can assure the order in which interfaces are detected (and indexed) at boot time, therefore, no assumption of the order should be assumed.
  It should be noted that this is only relevant for multi interfaces scenarios and not for the most common case of one interface per VM.
  - The interface naming is an old known issue which has been resolved by the consistent/predictable network device naming [1][2].
  - The default cloud images from some of the distributions have the consistent naming disabled (probably because most assume a single interface setup).
    (if one uses such an image, look at the grub config for the kernel command line and remove the `net.ifnames=0` specification)

The resolution of the issue reported in the BZ is to enable the consistent network device naming in order to avoid dependency on the interface detection order.


1. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/ch-consistent_network_device_naming#sec-Naming_Schemes_Hierarchy
2. https://www.freedesktop.org/software/systemd/man/systemd.net-naming-scheme.html

(In reply to Edward Haas from comment #14)
[...]
> 
> The resolution of the issue reported in the BZ is to enable the consistent
> network device naming in order to avoid dependency on the interface
> detection order.

With current QEMU, consistent network device naming will use either
slot or path based naming scheme. It should work in most cases assuming
immutable XML. However if PCI topology changes or machine type changes,
previously used interface name may change as well.

consistent network device naming also supports 'onboard' naming scheme,
in which case for naming interfaces it uses 'eno' prefix with acpi_index
supplied by firmware.
So if there is interest, it's possible to add missing ACPI code to QEMU
and let user to specify acpi_index for a network card. That way user should be
able to control interface naming and avoid name changes even if machine type
flips between PC|Q35 or PCI topology changes.
(as minimum, it should work for cold-plugged PCI devices)

Igor - now that I've digested what you've said, this sounds like a potentially *very good* thing to have. It would require a proper configuration to be provided by management (so you wouldn't be able to take some random cloud image and boot it up with "just any old config" and have the device named correctly; but at least it would be possible to have a guaranteed name for a device without depending on a specific MAC address (which would mean that the image would need to be modified for each instance).

Count me in if you want to do any experimenting with this idea!