RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2097270 - fix order/priority for IPv6 addresses configured by NetworkManager in [rhel-8.7]
Summary: fix order/priority for IPv6 addresses configured by NetworkManager in [rhel-8.7]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: NetworkManager
Version: 8.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: NetworkManager Development Team
QA Contact: Vladimir Benes
Jaroslav Klech
URL:
Whiteboard:
: 2073032 (view as bug list)
Depends On:
Blocks: 2079851 2097293
TreeView+ depends on / blocked
 
Reported: 2022-06-15 10:20 UTC by Thomas Haller
Modified: 2022-11-21 11:32 UTC (History)
7 users (show)

Fixed In Version: NetworkManager-1.39.7-2.el8
Doc Type: Bug Fix
Doc Text:
.The `NetworkManager` utility enforces correct ordering of IPv6 addresses from various sources In general, the ordering of IPv6 addresses affects the priority for source address selection. For example, when you make an outgoing TCP connection. Previously, the relative priority of IPv6 addresses added through the `manual`, `dhcpv6`, and `autoconf6` methods, was not correct. With this update, the problem has been fixed and the ordering priority now reflects this logic: `manual` > `dhcpv6` > `autoconf6`. However, the order of addresses under the `ipv6.addresses` setting did not change and the address added last still has the highest priority.
Clone Of:
: 2097293 (view as bug list)
Environment:
Last Closed: 2022-11-08 10:10:38 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-125328 0 None None None 2022-06-15 10:25:29 UTC
Red Hat Product Errata RHBA-2022:7680 0 None None None 2022-11-08 10:12:04 UTC

Description Thomas Haller 2022-06-15 10:20:35 UTC
Background
----------

The priority of IP addresses matters, especially for source address selection.

In kernel, the priority cannot directly configured, instead it depends on the order in which addresses are added.

For IPv4, we (and kernel) only care about the relative priority of addresses in the same subnet. Here, addresses added first, will become the primary address, while further addresses (from the same subnet!) get the "secondary" flag. Addresses from different subnet have undefined priority, as far as NetworkManager is concerned.

For IPv6, the priority only matters for addresses of the same scope. This means all global addresses need to have a priority determined. In kernel, the IPv6 address added last, will get the highest priority. The priority corresponds to the order in which you see addresses in `ip -6 addr`.


This means, NetworkManager has an intended priority, and when it configures addresses, it needs to add them in the right order. This also means, it may be necessary for NetworkManager to remove existing addresses, to re-add them in the correct order.


Details and current state
-------------------------

There are two CI tests that test the order. See @ipv6_check_addr_order and @ipv4_check_addr_order. See them on https://tools.dqe.lab.eng.bos.redhat.com/vbenes/nm_ci_stats/stats.html#

For IPv4, rhel-8.7/rhel-9.1 follows latest upstream behavior and is correct.

Before 1.36 (rhel-8.5 and older) there was "some behavior". As the code was hard to understand and we didn't have sufficient CI coverage (or even understanding what exactly should happen), we broke that in 1.36 (rhel-8.6, rhel-9.0). Check the CI tests in the ci-states for details about how the tests show the behavior.

In 1.36.0, there are actually two bugs. One is that NetworkManager now wants to configure IPv6 addresses in reversed order.  The second bug is that it won't remove addresses to fix the order, so the order is effectively just the order in which addresses got added.

For IPv6, (time-wise) we would add SLAAC addresses before DHCPv6 (because they appear first). Due to bug 1, we would want to prefer SLAAC over DHCPv6 (wrongly), but due to bug 2, it didn't actually happen. That is actually good, because users probably want that DHCPv6 is preferred over SLAAC addresses, which was also in 8.5.

This is all fixed in 1.38.0 and newer.

In 1.36.5, we backported a fix for bug 2 ([1]), but that caused bug 1 to hit. Note that fixing bug 2 was correct for IPv4, but problematic for IPv6 with bug 1. The solution for 1.36.7 was to backport all relevant fixes from nm-1-38. We changed behavior there, but upstream behavior will be good with the next 1.36.8 release ([2]).

for rhel-8.6 and rhel-9.0 Z steam updates, we included the patch for fixing bug 2 ([1]). That is good for IPv4, but as we didn't do anything about bug 1, we also added a followup fix which disables the change for IPv6. So, rhel-8.6 and rhel-9.0 basically still has the problem that the IPv6 address order is not enforced, and if it were enforced, it would be wrong. Probably we won't solve this for 8.6/9.0, because of the change in behavior that the bugfix brings :(

rhel-8.7 and rhel-9.1 is based on latest upstream version. It gets all the bugfixes, however, we included a downstream patch ([3]) that tries to get closer to the 8.6/9.0 behavior.

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/cd4601802de52f0239b5b8c19bd90ed9ccb6a50c
[2] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/6920b6ceb1c2e7856ad76e118ee5b4dd36130735
[3] http://pkgs.devel.redhat.com/cgit/rpms/NetworkManager/tree/0003-order-ipv6-addresses.patch?id=62a7f737acc4536755861f768f7d5fe276e3010d

However, the behavior of 8.6/9.0 is just too wrong, and it was already different compared to rhel-8.5. The only sensible way forward is now to drop the downstream patch from 8.7/9.0, document the potential change due to the bugfix, and follow upstream behavior. This actually gets closer to rhel-8.5 behavior.


Solution
--------

The solution is to drop downstream patch 0003-order-ipv6-addresses.patch for 8.7/9.1 and follow upstream behavior.

That downstream patch (and it's removal) changes two things:

- the order of addresses for different sources. That is, SLAAC vs. DHCPv6 vs. manual addresses. The current notion that SLAAC is to be preferred over DHCPv6 is just wrong. Even if rhel-8.6 partly has that notion, we need to change that.

- the order of manual addresses in "ipv6.addresses". On upstream `main` the first address in the list will be preferred, previously it was the last. I would also do this change, as part of the bugfix, because the new behavior makes more sense. Also, because who even configures static IPv6 addresses, and who configures more of one of them, and who then also cares about their priority? I'd hope this would affect very few people, but as we already had broken behavior in 8.6, I would take this opportunity to fix it once and for all.

Comment 1 Thomas Haller 2022-06-15 10:47:04 UTC
IPv6 source address selection is defined in rfc6724, and that is what kernel implements

https://datatracker.ietf.org/doc/html/rfc6724

It lists 8 rules for sorting/comparing addresses to find a best one.

However, in case of a tie, the RFC says:

  If the eight rules fail to choose a single address, the tiebreaker is implementation-specific.

This is what we are talking about, and where the order of IP addresses becomes relevant.

Comment 2 Thomas Haller 2022-06-15 11:30:12 UTC
Alternative solutions are of course to configure static routes with "src" attribute.

For DHCPv4, we will now automatically set the "src" attribute for routes obtained via DHCP. However, DHCPv6 does not provide routes, so this is only a solution for static addresses. 

It still seems kinda important to prefer DHCPv6 over SLAAC. See also the linked bugs in https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/6920b6ceb1c2e7856ad76e118ee5b4dd36130735

Comment 3 Beniamino Galvani 2022-06-15 12:00:36 UTC
(In reply to Thomas Haller from comment #0)

> - the order of manual addresses in "ipv6.addresses". On upstream `main` the
> first address in the list will be preferred, previously it was the last. I
> would also do this change, as part of the bugfix, because the new behavior
> makes more sense. Also, because who even configures static IPv6 addresses,
> and who configures more of one of them, and who then also cares about their
> priority? I'd hope this would affect very few people, but as we already had
> broken behavior in 8.6, I would take this opportunity to fix it once and for
> all.

I agree with the plan, but I'm not sure about changing the quoted part. While
for DHCPv6 vs SLAAC vs static there are some addresses that should have more
priority than others, for static addresses the order is arbitrarily decided by NM.
I wouldn't change this convention in a minor RHEL8 release just because upstream
did it.

Comment 4 Thomas Haller 2022-06-15 13:28:03 UTC
(In reply to Beniamino Galvani from comment #3)
> (In reply to Thomas Haller from comment #0)

> I agree with the plan, but I'm not sure about changing the quoted part. While
> for DHCPv6 vs SLAAC vs static there are some addresses that should have more
> priority than others, for static addresses the order is arbitrarily decided
> by NM.
> I wouldn't change this convention in a minor RHEL8 release just because
> upstream
> did it.

OK, then lets keep that aspect in rhel-8.7+.
What about rhel-9.1+?

Comment 5 Beniamino Galvani 2022-06-15 13:33:10 UTC
> OK, then lets keep that aspect in rhel-8.7+.

Sounds good.

> What about rhel-9.1+?

For RHEL 9 I think it's fine to adopt the upstream behavior.

Comment 6 Thomas Haller 2022-06-15 13:36:11 UTC
(In reply to Beniamino Galvani from comment #5)
> For RHEL 9 I think it's fine to adopt the upstream behavior.

See: https://bugzilla.redhat.com/show_bug.cgi?id=2097293#c6

Comment 10 Vladimir Benes 2022-07-12 13:04:22 UTC
we still preserve reversed order of IPv6 addresses, but addresses are strictly kept that way and not changing anyhow when we send them in different order later on, we will keep this in rhel8.x
https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/blob/main/features/scenarios/ipv6.feature#L1003

Comment 12 Thomas Haller 2022-08-31 17:50:01 UTC
doctext lgtm, thanks Jaroslav.

If I had a tiny complaint, from "However, the order of addresses under the `ipv6.addresses` setting was not reversed and the address added last has the highest priority." it sounds not very clear that this is a no-change. It was like this, and didn't change (although we changed in rhel-9.1+ and upstream). But good as-is.

Comment 13 Jaroslav Klech 2022-09-01 08:13:45 UTC
(In reply to Thomas Haller from comment #12)
> doctext lgtm, thanks Jaroslav.
> 
> If I had a tiny complaint, from "However, the order of addresses under the
> `ipv6.addresses` setting was not reversed and the address added last has the
> highest priority." it sounds not very clear that this is a no-change. It was
> like this, and didn't change (although we changed in rhel-9.1+ and
> upstream). But good as-is.

Thanks Thomas,

i updated the text in this bugzilla based on your feedback.

I am assuming that the text in bz#2097293 (9.1 counterpart) looks good to you as well?

Regards,
Jaroslav

Comment 14 Thomas Haller 2022-09-08 15:18:23 UTC
(In reply to Jaroslav Klech from comment #13)
> i updated the text in this bugzilla based on your feedback.

lgtm

Comment 16 errata-xmlrpc 2022-11-08 10:10:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7680

Comment 17 Thomas Haller 2022-11-21 11:32:10 UTC
*** Bug 2073032 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.