Hide Forgot
Background ---------- The priority of IP addresses matters, especially for source address selection. In kernel, the priority cannot directly configured, instead it depends on the order in which addresses are added. For IPv4, we (and kernel) only care about the relative priority of addresses in the same subnet. Here, addresses added first, will become the primary address, while further addresses (from the same subnet!) get the "secondary" flag. Addresses from different subnet have undefined priority, as far as NetworkManager is concerned. For IPv6, the priority only matters for addresses of the same scope. This means all global addresses need to have a priority determined. In kernel, the IPv6 address added last, will get the highest priority. The priority corresponds to the order in which you see addresses in `ip -6 addr`. This means, NetworkManager has an intended priority, and when it configures addresses, it needs to add them in the right order. This also means, it may be necessary for NetworkManager to remove existing addresses, to re-add them in the correct order. Details and current state ------------------------- There are two CI tests that test the order. See @ipv6_check_addr_order and @ipv4_check_addr_order. See them on https://tools.dqe.lab.eng.bos.redhat.com/vbenes/nm_ci_stats/stats.html# For IPv4, rhel-8.7/rhel-9.1 follows latest upstream behavior and is correct. Before 1.36 (rhel-8.5 and older) there was "some behavior". As the code was hard to understand and we didn't have sufficient CI coverage (or even understanding what exactly should happen), we broke that in 1.36 (rhel-8.6, rhel-9.0). Check the CI tests in the ci-states for details about how the tests show the behavior. In 1.36.0, there are actually two bugs. One is that NetworkManager now wants to configure IPv6 addresses in reversed order. The second bug is that it won't remove addresses to fix the order, so the order is effectively just the order in which addresses got added. For IPv6, (time-wise) we would add SLAAC addresses before DHCPv6 (because they appear first). Due to bug 1, we would want to prefer SLAAC over DHCPv6 (wrongly), but due to bug 2, it didn't actually happen. That is actually good, because users probably want that DHCPv6 is preferred over SLAAC addresses, which was also in 8.5. This is all fixed in 1.38.0 and newer. In 1.36.5, we backported a fix for bug 2 ([1]), but that caused bug 1 to hit. Note that fixing bug 2 was correct for IPv4, but problematic for IPv6 with bug 1. The solution for 1.36.7 was to backport all relevant fixes from nm-1-38. We changed behavior there, but upstream behavior will be good with the next 1.36.8 release ([2]). for rhel-8.6 and rhel-9.0 Z steam updates, we included the patch for fixing bug 2 ([1]). That is good for IPv4, but as we didn't do anything about bug 1, we also added a followup fix which disables the change for IPv6. So, rhel-8.6 and rhel-9.0 basically still has the problem that the IPv6 address order is not enforced, and if it were enforced, it would be wrong. Probably we won't solve this for 8.6/9.0, because of the change in behavior that the bugfix brings :( rhel-8.7 and rhel-9.1 is based on latest upstream version. It gets all the bugfixes, however, we included a downstream patch ([3]) that tries to get closer to the 8.6/9.0 behavior. [1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/cd4601802de52f0239b5b8c19bd90ed9ccb6a50c [2] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/6920b6ceb1c2e7856ad76e118ee5b4dd36130735 [3] http://pkgs.devel.redhat.com/cgit/rpms/NetworkManager/tree/0003-order-ipv6-addresses.patch?id=62a7f737acc4536755861f768f7d5fe276e3010d However, the behavior of 8.6/9.0 is just too wrong, and it was already different compared to rhel-8.5. The only sensible way forward is now to drop the downstream patch from 8.7/9.0, document the potential change due to the bugfix, and follow upstream behavior. This actually gets closer to rhel-8.5 behavior. Solution -------- The solution is to drop downstream patch 0003-order-ipv6-addresses.patch for 8.7/9.1 and follow upstream behavior. That downstream patch (and it's removal) changes two things: - the order of addresses for different sources. That is, SLAAC vs. DHCPv6 vs. manual addresses. The current notion that SLAAC is to be preferred over DHCPv6 is just wrong. Even if rhel-8.6 partly has that notion, we need to change that. - the order of manual addresses in "ipv6.addresses". On upstream `main` the first address in the list will be preferred, previously it was the last. I would also do this change, as part of the bugfix, because the new behavior makes more sense. Also, because who even configures static IPv6 addresses, and who configures more of one of them, and who then also cares about their priority? I'd hope this would affect very few people, but as we already had broken behavior in 8.6, I would take this opportunity to fix it once and for all.
IPv6 source address selection is defined in rfc6724, and that is what kernel implements https://datatracker.ietf.org/doc/html/rfc6724 It lists 8 rules for sorting/comparing addresses to find a best one. However, in case of a tie, the RFC says: If the eight rules fail to choose a single address, the tiebreaker is implementation-specific. This is what we are talking about, and where the order of IP addresses becomes relevant.
Alternative solutions are of course to configure static routes with "src" attribute. For DHCPv4, we will now automatically set the "src" attribute for routes obtained via DHCP. However, DHCPv6 does not provide routes, so this is only a solution for static addresses. It still seems kinda important to prefer DHCPv6 over SLAAC. See also the linked bugs in https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/6920b6ceb1c2e7856ad76e118ee5b4dd36130735
(In reply to Thomas Haller from comment #0) > - the order of manual addresses in "ipv6.addresses". On upstream `main` the > first address in the list will be preferred, previously it was the last. I > would also do this change, as part of the bugfix, because the new behavior > makes more sense. Also, because who even configures static IPv6 addresses, > and who configures more of one of them, and who then also cares about their > priority? I'd hope this would affect very few people, but as we already had > broken behavior in 8.6, I would take this opportunity to fix it once and for > all. I agree with the plan, but I'm not sure about changing the quoted part. While for DHCPv6 vs SLAAC vs static there are some addresses that should have more priority than others, for static addresses the order is arbitrarily decided by NM. I wouldn't change this convention in a minor RHEL8 release just because upstream did it.
(In reply to Beniamino Galvani from comment #3) > (In reply to Thomas Haller from comment #0) > I agree with the plan, but I'm not sure about changing the quoted part. While > for DHCPv6 vs SLAAC vs static there are some addresses that should have more > priority than others, for static addresses the order is arbitrarily decided > by NM. > I wouldn't change this convention in a minor RHEL8 release just because > upstream > did it. OK, then lets keep that aspect in rhel-8.7+. What about rhel-9.1+?
> OK, then lets keep that aspect in rhel-8.7+. Sounds good. > What about rhel-9.1+? For RHEL 9 I think it's fine to adopt the upstream behavior.
(In reply to Beniamino Galvani from comment #5) > For RHEL 9 I think it's fine to adopt the upstream behavior. See: https://bugzilla.redhat.com/show_bug.cgi?id=2097293#c6
we still preserve reversed order of IPv6 addresses, but addresses are strictly kept that way and not changing anyhow when we send them in different order later on, we will keep this in rhel8.x https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/blob/main/features/scenarios/ipv6.feature#L1003
doctext lgtm, thanks Jaroslav. If I had a tiny complaint, from "However, the order of addresses under the `ipv6.addresses` setting was not reversed and the address added last has the highest priority." it sounds not very clear that this is a no-change. It was like this, and didn't change (although we changed in rhel-9.1+ and upstream). But good as-is.
(In reply to Thomas Haller from comment #12) > doctext lgtm, thanks Jaroslav. > > If I had a tiny complaint, from "However, the order of addresses under the > `ipv6.addresses` setting was not reversed and the address added last has the > highest priority." it sounds not very clear that this is a no-change. It was > like this, and didn't change (although we changed in rhel-9.1+ and > upstream). But good as-is. Thanks Thomas, i updated the text in this bugzilla based on your feedback. I am assuming that the text in bz#2097293 (9.1 counterpart) looks good to you as well? Regards, Jaroslav
(In reply to Jaroslav Klech from comment #13) > i updated the text in this bugzilla based on your feedback. lgtm
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:7680
*** Bug 2073032 has been marked as a duplicate of this bug. ***