Bug 2211896

I think the bug is duplicated with bug1642021

The root cause is in my opinion similar. The difference is that the guest does not contain old system scripts but relies on NetworkManager. If the .nmconnection file is edited and "interface-name" key removed the wired connection comes up with DHCP. I did not investigate if it's possible to inject MAC instead of interface name into the .nmconnection file.

I have found out that it is possible to specify MAC (mac-address) instead of interface (interface-name) in nmconnection file. But then I also realized that this is not really helpful because virt-v2v does not even know the mapping between MAC and inteface name on source.

(In reply to Tomáš Golembiovský from comment #3)
> But then I also realized
> that this is not really helpful because virt-v2v does not even know the
> mapping between MAC and inteface name on source.

Yeah, not for an automated solution but it sounds like a possible workaround (i.e., telling users to change it in the guest before starting the migration)

(In reply to Tomáš Golembiovský from comment #3)
> I have found out that it is possible to specify MAC (mac-address) instead of
> interface (interface-name) in nmconnection file. But then I also realized
> that this is not really helpful because virt-v2v does not even know the
> mapping between MAC and inteface name on source.

We use the following method to let rhel9 guests get IP from DHCP if network name is changed:

In my example, the rhel9 guest's network name was changed from ens192 to enp1s0 after v2v conversion

1. After v2v conversion, boot and log in to the guest, and copy ens192.nmconnection to enp1s0.nmconnection in /etc/NetworkManager/system-connections

2. Modify enp1s0.nmconnection and change the id to enp1s0

3. Restart NetworkManager service, then rhel9 guest can get IP from DHCP
#systemctl restart NetworkManager

(In reply to mxie from comment #5)
> (In reply to Tomáš Golembiovský from comment #3)
> > I have found out that it is possible to specify MAC (mac-address) instead of
> > interface (interface-name) in nmconnection file. But then I also realized
> > that this is not really helpful because virt-v2v does not even know the
> > mapping between MAC and inteface name on source.
> 
> We use the following method to let rhel9 guests get IP from DHCP if network
> name is changed:
> 
> In my example, the rhel9 guest's network name was changed from ens192 to
> enp1s0 after v2v conversion
> 
> 1. After v2v conversion, boot and log in to the guest, and copy
> ens192.nmconnection to enp1s0.nmconnection in
> /etc/NetworkManager/system-connections
> 
> 2. Modify enp1s0.nmconnection and change the id to enp1s0
> 
> 3. Restart NetworkManager service, then rhel9 guest can get IP from DHCP
> #systemctl restart NetworkManager

The downside of this approach is twofold:
1. It means that qemu-guest-agent is not installed during the migration, right?
2. It doesn't scale well (what I wrote in comment 4 also doesn't scale well)

(In reply to Arik from comment #6) 
> The downside of this approach is twofold:
> 1. It means that qemu-guest-agent is not installed during the migration,
> right?
> 2. It doesn't scale well (what I wrote in comment 4 also doesn't scale well)

We will configure guest's network name before v2v conversion, therefore, the installation of qemu-guest-agent is not affected after v2v conversion

In <https://bugzilla.redhat.com/show_bug.cgi?id=1642021#c12> I wrote "this problem is practically unsolvable", and provided a link to <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/consistent-network-interface-device-naming_configuring-and-managing-networking> (and set Devel Cond-NAK: Design on that bug).

When you convert a a guest from vmw to qemu/kvm, the virtual motherboard changes, the PCI addresses change, the guest firmware changes. Per "consistent network interface device naming" (see the link above), these changes are bound to change the device name that schemes 1 through 3 derive for the NIC. Only scheme 4 uses the MAC address ("Red Hat Enterprise Linux does not use this scheme by default"). Virt-v2v does not deal with PCI(e) addresses on either the source or the destination host, so we can't control the new name.

"Consistent network interface device naming" is a great concept *if* your focus is on the stability of a single computer; that is, if you have a NIC in place, and add further NICs, or remove NICs *other* than the subject NIC. Then the consistent naming guarantees your particular subject NIC will see no regressions.

"Consistent network interface device naming" is a terrible concept *if* your focus is on *moving* a particular NIC from one computer to another. In the other computer, all the physical locations (PCIe addresses and so on) will be different. Because the naming is tied to the physical location of the slot that accepts the card, and not the card's properties (MAC address), there's not going to be any consistency or compatibility. That will only work if you switch the OS to scheme#4 (MAC-based naming) first, I think.

Tangentially, see also guestfs-tools BZ#1980922.

So my view on this bug is the management system ought to preserve the MAC address
from the source, and NM ought to recognize that and assign the same mapping when
the guest boots.  Now it seems like you are doing this (good!) and NM is still not
bringing up the interface, so probably this is an NM bug.

The interface renaming is something of a separate issue where I think that the
whole thing is a stupid mess and it would have been better to stick with eth0, eth1 etc.

Nevertheless NM ought to deal with this for us.

Advance warning - this comment contains no useful solutions or workarounds! I was just curious about this "ens" naming, and investigate a bit on how it might happen...)

When net.ifnames=1 (which has been the standard for several years now), interfaces normally have names prefixed with "enp", followed by information about the PCI bus/slot/function where the device is connected.

In recent releases of libvirt+QEMU, you can add <acpi index='N'/> to the interface definition, and on a Q35 machinetype the interface will end up instead being prefixed with "eno" and followed by the index provided in the config, (in this example, "eno4").

I looked up what would cause an interface to be named "ensN", and found this page:

  https://systemd.io/PREDICTABLE_INTERFACE_NAMES/

It shows the origins of eno and ens to be very similar:

  1. Names incorporating Firmware/BIOS provided index numbers for on-board devices (example: eno1)
  2. Names incorporating Firmware/BIOS provided PCI Express hotplug slot index numbers (example: ens1)

I did some experimenting and found that to be "kind of" correct, but not really.

In the end, the only way I could get a network device named "ens192" was to create a 440fx machinetype, and add <acpi index='192'/> to the interface spec. Here are the things I tried:

For Q35 machinetype:

A) add "<acpi index='192'/> to an interface  --> eno192
B) add "<acpi index='192'/> to an interface connected to a conventional PCI slot --> enp15s2
C) Do (A) and (B) with ACPI hotplug disabled:   A) enp10s0, B) ens2 (so close!)
D) hotplug a network device to a running system --> no change in any case

(NB: if you wonder how changing the "port" and "chassis" numbers of the interface's pcie-root-port affect the naming, the answer is "they don't")

For 440fx machinetype:

A) add "<acpi index='192'/> to interface --> ens192

So it looks like getting the "ens" naming on a Q35 machinetype requires disabling ACPI hotplug (which we've repeatedly avoided) *AND* plugging the device into a conventional PCI slot, which we've explicitly and repeatedly decided we don't want to do. (Also we would then need to plug the device into "slot 192" in order to get the name desired in this particular case, and of course that isn't possible because slot numbers must be between 0 and 31).

TL;DR - the solution is going to have to involve modifying the config of the guest one way or another. If the guest config contains the MAC address and previous interface name, then v2v could maybe look for devices named "ensBLAH", changed their names in the guest config to "enoBLAH", and add <acpi index='BLAH'/> to the interface config.

Sigh.

In my haste, I misread the 440fx "ip link" output - the device there is *also* named "eno192".

The "ens" devices on 440fx were all of those that didn't have an acpi index specified. The one that *did* have <acpi index='192'/> showed up just as it did on the Q35 machine.

Anyway, the TL;DR remains the same.

(In reply to Laine Stump from comment #10)
> Advance warning - this comment contains no useful solutions or workarounds!
> I was just curious about this "ens" naming, and investigate a bit on how it
> might happen...)
> 
> When net.ifnames=1 (which has been the standard for several years now),
> interfaces normally have names prefixed with "enp", followed by information
> about the PCI bus/slot/function where the device is connected.
> 
> In recent releases of libvirt+QEMU, you can add <acpi index='N'/> to the
> interface definition, and on a Q35 machinetype the interface will end up
> instead being prefixed with "eno" and followed by the index provided in the
> config, (in this example, "eno4").
> 
> I looked up what would cause an interface to be named "ensN", and found this
> page:
> 
>   https://systemd.io/PREDICTABLE_INTERFACE_NAMES/
> 
> It shows the origins of eno and ens to be very similar:
> 
>   1. Names incorporating Firmware/BIOS provided index numbers for on-board
> devices (example: eno1)

don't be scared off with 'onboard' language,
it can be used for pluggable NICs as long as platform is able to
describe it in ACPI tables.

QEMU wise, this schema (acpi-index) was introduced 1st on PC machines in 6.0
since it already had ACPI PCI hotplug infrastructure describing PCI slots
(which happens to include slots on host bridge) and Daniel complemented it
with libvirt support.

In the next release (6.1), Q35 machine got ACPI PCI hotplug and with it
inherited acpi-index (caveat: Q35 host-bridge does not support hotplug =>
NICs directly attached to it don't have acpi-index support). Any other nic
attached to root-ports or pci bridges shall have working acpi-index.

Since 8.0, acpi-index was decoupled from ACPI hotplug and should work
for non-hotpluggble NICs as well (ex: nics directly attached to q35 host bridge,
or to present at boot root-ports/bridges even if ACPI hotplug has been disabled)
See QEMU's Changelog for details.

What's not been implemented yet is acpi-index support on PXBs (work in progress)
and there are plans to make it work for arm-virt (by unifying ACPI PCI code
with x86 one)

>   2. Names incorporating Firmware/BIOS provided PCI Express hotplug slot
> index numbers (example: ens1)

QEMU provides _SUN object per slot when ACPI hotplug is enabled which
can be used by this schema. This should have impact on both Q35 and PC
machines since ACPI hotplug is default. Though schema is still PCI topology
depended. So unless v2v reimplements systemd's renaming impl. it won't be
able to guess the name.

[...]
> TL;DR - the solution is going to have to involve modifying the config of the
> guest one way or another. If the guest config contains the MAC address and
> previous interface name, then v2v could maybe look for devices named
> "ensBLAH", changed their names in the guest config to "enoBLAH", and add
> <acpi index='BLAH'/> to the interface config.

I kind of agree with Laine on this, v2v or p2v for that matter
has to modify guest network configuration one way or another. With
'predictable network interface names' deployed out there for many years
it is better to use 'enoX' schema by assigning acpi-index to NICs as
it gives direct control over NIC naming from mgmt side without need
to deal with PCI topology nuances.

(In reply to Laine Stump from comment #10)

> If the guest config contains the MAC address and previous interface
> name, then v2v could maybe look for devices named "ensBLAH", changed
> their names in the guest config to "enoBLAH", and add <acpi
> index='BLAH'/> to the interface config.

We need three links, between four layers:

  src guest -[1]-> src VMM -[2]-> dst VMM -[3]-> dst guest

Virt-v2v's assumption is (more or less) that the MAC address is usable
for all three links. It handles link [2] explicitly, and assumes the
guest (links [1] and [3]) to be inherently MAC-based.

I understand this proposal to be "acpi-index" for the dst VMM (which is
not easy in itself, because when will RHV grow support for that, in
their OVF format?), and "enoX" for the dst guest. Then link [3] would be
covered, without depending on the MAC address.

Link [2] would be implemented in virt-v2v (the conversion logic).
Virt-v2v could take e.g. the src domain XML (as dumped from vmware by
libvirt; comment#0):

    <interface type='bridge'>
      <mac address='00:50:56:bb:39:aa' type='generated'/>
      <source bridge='Mgmt Network'/>
      <model type='e1000'/>
    </interface>

and augment that (beyond the tweaks we're already doing) with
"acpi-index".

However, link [1] seems to be missing (i.e., where Laine writes "[i]f
the guest config contains the MAC address *AND* previous interface
name"; emphasis mine).

On my RHEL-9 laptop, I have

  /etc/NetworkManager/system-connections/enp9s0u1u4u4u3.nmconnection

containing

  [connection]
  id=enp9s0u1u4u4u3
  uuid=29d3f6f8-8ce7-3e03-bf6e-65a97d694bec
  type=ethernet
  autoconnect-priority=-999
  interface-name=enp9s0u1u4u4u3
  timestamp=1670342568

  [ethernet]

  [ipv4]
  method=auto

  [ipv6]
  addr-gen-mode=eui64
  method=auto

  [proxy]

making in total for three instances of "enp9s0u1u4u4u3" (filename, id,
interface-name), and *zero* instances of the MAC address.

In other words, we cannot connect the source guest information (such as
"ens192") to the source VMM information (the MAC address).

In particular, the <interface> element dumped by libvirt about a vmware
source guest only contains a MAC, and no PCI topology or ACPI
information -- so even if we duplicated systemd's logic in virt-v2v, we
wouldn't have the input data for that algorithm.

Assume we have two NICs in the source guest, "ens192" and "ens193" (just
making up the names), two corresponding *.nmconnection files (not
mentioning MAC addresses), and two <interface> elements with nothing but
MAC addresses. We might be able to rename and update the *.nmconnection
files (ens192->eno192, ens193->eno193), but *which* <interface> element
of the two do we insert <acpi index='192'/> in, as opposed to <acpi
index='193'/>? The source <interface> elements are only identifiable by
MAC address, and the source guest *.nmconnection files don't tie "ensX"
to any MAC address.

(In reply to Laszlo Ersek from comment #13)
> (In reply to Laine Stump from comment #10)
> 
> > If the guest config contains the MAC address and previous interface
> > name, then v2v could maybe look for devices named "ensBLAH", changed
> > their names in the guest config to "enoBLAH", and add <acpi
> > index='BLAH'/> to the interface config.
> 
> We need three links, between four layers:
> 
>   src guest -[1]-> src VMM -[2]-> dst VMM -[3]-> dst guest
> 
> Virt-v2v's assumption is (more or less) that the MAC address is usable
> for all three links. It handles link [2] explicitly, and assumes the
> guest (links [1] and [3]) to be inherently MAC-based.

[1] and [3] depends on what guest is and how guest is enumerating
interfaces. For example: MAC address + the same place in PCI topology
may be not enough. (Windows guest may re-enumerate/create a new interface
depending on QEMU/machine version & virtio drivers version)  

> I understand this proposal to be "acpi-index" for the dst VMM (which is
> not easy in itself, because when will RHV grow support for that, in
> their OVF format?), and "enoX" for the dst guest. Then link [3] would be
> covered, without depending on the MAC address.

We should ask RHV/cloud folks to look into it
(there were some interest from them some time ago, but I no longer
recall who that was). Anyways I can try to find out who to bother
with it.

> Link [2] would be implemented in virt-v2v (the conversion logic).
> Virt-v2v could take e.g. the src domain XML (as dumped from vmware by
> libvirt; comment#0):
> 
>     <interface type='bridge'>
>       <mac address='00:50:56:bb:39:aa' type='generated'/>
>       <source bridge='Mgmt Network'/>
>       <model type='e1000'/>
>     </interface>
> 
> and augment that (beyond the tweaks we're already doing) with
> "acpi-index".
> 
> However, link [1] seems to be missing (i.e., where Laine writes "[i]f
> the guest config contains the MAC address *AND* previous interface
> name"; emphasis mine).
> 
> On my RHEL-9 laptop, I have
> 
>   /etc/NetworkManager/system-connections/enp9s0u1u4u4u3.nmconnection
> 
> containing
> 
>   [connection]
>   id=enp9s0u1u4u4u3
>   uuid=29d3f6f8-8ce7-3e03-bf6e-65a97d694bec
>   type=ethernet
>   autoconnect-priority=-999
>   interface-name=enp9s0u1u4u4u3
>   timestamp=1670342568
> 
>   [ethernet]
> 
>   [ipv4]
>   method=auto
> 
>   [ipv6]
>   addr-gen-mode=eui64
>   method=auto
> 
>   [proxy]
> 
> making in total for three instances of "enp9s0u1u4u4u3" (filename, id,
> interface-name), and *zero* instances of the MAC address.
> 
> In other words, we cannot connect the source guest information (such as
> "ens192") to the source VMM information (the MAC address).
> 
> In particular, the <interface> element dumped by libvirt about a vmware
> source guest only contains a MAC, and no PCI topology or ACPI
> information -- so even if we duplicated systemd's logic in virt-v2v, we
> wouldn't have the input data for that algorithm.
>
> Assume we have two NICs in the source guest, "ens192" and "ens193" (just
> making up the names), two corresponding *.nmconnection files (not
> mentioning MAC addresses), and two <interface> elements with nothing but
> MAC addresses. We might be able to rename and update the *.nmconnection
> files (ens192->eno192, ens193->eno193), but *which* <interface> element
> of the two do we insert <acpi index='192'/> in, as opposed to <acpi
> index='193'/>? The source <interface> elements are only identifiable by
> MAC address, and the source guest *.nmconnection files don't tie "ensX"
> to any MAC address.

I don't see any reliable way to deal with it, modulo 'live' migration
when src guest still runs on platform it's migrated from and where
one can inspect 'ip link' & co directly.

(a very unreliable way, could be looking for missing data in boot logs,
on my system I see NIC driver dumping MAC which one could
eventually chase down to systemd renamed interface name, but then
not every NIC driver would do that)

Maybe in case of multiple connections, ask a user to provide mapping?
or ask user to turn on arbitrary automatic re-mapping if one doesn't
care (which is very unlikely in case multiple NICs src)

PS:
it's likely no possible to emulate systemd's renaming scheme without
running it on src as it can be PCI topo depended ending up somewhere
at USB interface (like with laptop example above).

Thanks for investigating.

I think at this point we might at best document a manual workaround (cf. mxie's comment#5) in a KB article. (But even having this bug open to the public should help already.)

(In reply to mxie from comment #5)
> (In reply to Tomáš Golembiovský from comment #3)
> > I have found out that it is possible to specify MAC (mac-address) instead of
> > interface (interface-name) in nmconnection file. But then I also realized
> > that this is not really helpful because virt-v2v does not even know the
> > mapping between MAC and inteface name on source.
> 
> We use the following method to let rhel9 guests get IP from DHCP if network
> name is changed:
> 
> In my example, the rhel9 guest's network name was changed from ens192 to
> enp1s0 after v2v conversion
> 
> 1. After v2v conversion, boot and log in to the guest, and copy
> ens192.nmconnection to enp1s0.nmconnection in
> /etc/NetworkManager/system-connections
> 
> 2. Modify enp1s0.nmconnection and change the id to enp1s0

Also need to change interface-name to enp1s0

> 3. Restart NetworkManager service, then rhel9 guest can get IP from DHCP
> #systemctl restart NetworkManager

(In reply to Laszlo Ersek from comment #15)
> Thanks for investigating.
> 
> I think at this point we might at best document a manual workaround (cf.
> mxie's comment#5) in a KB article. (But even having this bug open to the
> public should help already.)

Workaround is good in absence of real solution, but we probably can do better
for the case with single NIC by renaming it without caring about interface name
on source.

PS:
acpi-index support has been merged into KubeVirt API
http://kubevirt.io/api-reference/main/definitions.html#_v1_interface
not sure if any user facing tools picked it up though.