Bug 2079851

+++ This bug was initially created as a clone of Bug #2073032 +++

SUMMARY:
========

The order/priority how NM configures IPv6 is wrong. This should change, which is a visible change in behavior!!

- currently NM prefers autoconf6 addresses over DHCPv6 over manual. This is reversed. This seems a problem, because if you configure a manual/DHCPv6 address, it would stand to reason that you'd want to use that instead of the autoconf6 generated one.
- manual addresses in `ipv6.addresses` have the order reversed, so that the last address is the highest priority one.

See also CI test @ipv6_check_addr_order ([1]).

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/blob/b72d48bef8acd54e83e51ed668d8a0be84985bd7/features/scenarios/ipv6.feature#L1613).


BACKGROUND
==========

When investigating bug 1995372, I was looking at IPv4/IPv6 address order/priority.

This matters mostly for source address selection in kernel (see [2]).

[2] http://linux-ip.net/html/routing-saddr-selection.html

Kernel first checks the "src" attribute of the chosen route, and uses that as source address. In absence of these, kernel will pick an address, based on the order in which they were added (and the order in which you see them in `ip addr` output).

Btw, changing the priority of an address in kernel is very cumbersome. NetworkManager needs to remove all conflicting addresses on an interface and re-add them in the correct order. This can hardly be called an API, and requires considerable complexity to get right (because NetworkManager only wants to do this when necessary). This is done in nm_platform_ip_address_sync() function at [3].

[3] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/1234e5583a09a618b930241a12b8a1d580493568/src/libnm-platform/nm-platform.c#L3967

Note that [3] is without important bugfixes from mr1168 ([4]). The proper version of nm_platform_ip_address_sync() you find there.



NetworkManager thus cares about the order of addresses. At least partly.

- for IPv4, addresses that are in the same subnet, kernel will set the "secondary" flag. NM thus cares only about the relative order for addresses of the same subnet.

- for IPv6, I think the order only matters for addresses of the same scope. Thus, NetworkManager also here sorts addresses only within the same scope.



For IPv4, NetworkManager prefers manual addresses (from the profile) over DHCPV4. That is intentional. Bug 1995372 was asking to reverse that, however the proper solution for that is that DHCPv4 routes get the "src" attribute set. Also, bug 1995372 was concerned about the order of addresses of different subnets, which NetworkManager currently leaves undefined altogether. Anyway, bug 1995372 is solved differently, and IPv4 works IMO as desired and as implemented.

At least, it will be mostly fine with mr1168 ([4]). Before 1.36 we regressed this in subtle ways and there were other subtle bugs earlier too. Only mr#1168 will make everything right for IPv4.

[4] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1168

mr1168 [4] will also make it right for IPv6, in the sense that it does what is intentionally implemented. However, that is not what we really should do.




The Problem
===========


Kernel treats the IPv6 address that was added as last, as the most important one. Initially NetworkManager didn't care about the order of addresses, and thus when it tracked addresses in a list, it would configure it in reversed order (reversed in the sense on what one might reasonably expect).

When we started to care, we kept that reversed order. This is even documented in `man nm-settings-nmcli`, that `ipv6.addresses` has the highest priority address last.

We also merge addresses/DNS server/routes/etc from different sources (manual addresses, DHCPv6, autoconf6, link-local6). When we do that, we prefer information from manual over DHCPv6 over autoconf6 over link-local6 (as we similarly do for IPv4 case). However, as the list of addresses was in reversed order, this priority was got reversed.

That means, the priority is actually autoconf6 > DHCPv6 > manual-address (with the twist that manual addresses in "ipv6.addresses" are also tracked in reverse order).


I could live with the inconsistency of `ipv6.addresses` in the profile. But preferring autoconf addresses over DHCPv6 over manual addresses is just wrong. We need to fix that.


That fix is of course a change in behavior.



USE_TEMPADDR
============

One minor issue is "ipv6.ip6-privacy" setting (which is "use_tempaddr" sysctl). When that is enabled, NM will configure autoconf6 addresses with IFA_F_MANAGETEMPADDR, which causes kernel to add a "temporary" IPv6 address. The temporary addresss is added either right before or after the managing autoconf6 (depending on whether use_tempaddr sysctl is set to 1 or 2). That means, if we fix this, then we would configure first autoconf6 addresses (for which it adds the temporary addresses either before or after), then the DHCPv6/manual addresses would be added after -- which results in a higher priority. That's wrong.

So there is an exception to above rule. If "ipv6.ip6-privacy" is set to 2, then addresses with "IFA_F_MANAGETEMPADDR" will be preferred over all other addresses. So you'll have IFA_F_MANAGETEMPADDR address > manual > DHCPv6.


Note that NetworkManager can only configure the order of addresses by removing conflicting addresses and re-add them with the right priority. That means, if it first adds a IFA_F_MANAGETEMPADDR addresss, and later a DHCPv6 event comes, then it needs to remove the IFA_F_MANAGETEMPADDR address, then add the DHCPv6 address and re-add the autoconf6 address. When removing an address with IFA_F_MANAGETEMPADDR, kernel will not remove the temporary address (I guess that makes sense, temporary addresses should be deprecated and expire, and not removed). But that means, after this, you suddenly have 2 temporary addresses. And this reordering can repeat, so you get more temporary addresses. I don't know a nice solution to this, but it's probably good enough. The address will expire, and in IPv6 you must expect to have multiple addresses (especially with "use_tempaddr").

--- Additional comment from Thomas Haller on 2022-04-15 07:52:07 UTC ---

> USE_TEMPADDR
> ============

As Beniamino points out, there is probably no problem related to temporary addresses.
Kernel should prefer them: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1185#note_1339424

--- Additional comment from Thomas Haller on 2022-04-28 11:32:00 UTC ---

the actual change in behavior was merged upstream, with https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1185. Moving to POST.

Note that this change is potentially problematic material for RHEL-8.7.
Is it a desirable bug-fix, or an undesirable change in behavior? I argue it's the former. We need to document it properly.

There is a followup fix, which should help with certain edge cases: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1200