when reporting a bug against NetworkManager, please make an effort to provide level=TRACE logs.

See https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/main/contrib/fedora/rpm/NetworkManager.conf#L27 for hints about logging.

Thank you.

so I enabled trace, rebooted and this are the steps I did:

# uptime
 09:04:10 up 0 min,  1 user,  load average: 1.74, 0.43, 0.14
# cat /etc/sysconfig/network-scripts/ifcfg-ens3
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=10.94.0.5
PREFIX=24
GATEWAY=10.94.0.1
DNS1=192.168.1.42
DEFROUTE=yes
DHCP_VENDOR_CLASS_IDENTIFIER=anaconda-Linux
IPV4_FAILURE_FATAL=yes
IPV6INIT=yes
IPV6_AUTOCONF=no
IPV6ADDR_SECONDARIES="2a02:168:f00d:caab::2/64"
IPV6ADDR=2a02:168:f00d:caab:0:1::2/64
IPV6_DEFAULTGW=fe80::1
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=yes
NAME=ens3
UUID=0b4f730f-6894-43bc-942d-a23b2fd4dfea
DEVICE=ens3
ONBOOT=yes
MULTI_CONNECT=1
DEVTIMEOUT=60
# curl -6 icanhazip.com
2a02:168:f00d:caab::2
# systemctl restart NetworkManager
# date
Wed Aug  4 09:04:37 UTC 2021
# curl -6 icanhazip.com
2a02:168:f00d:caab:0:1:0:2
# journalctl --boot -u NetworkManager > NetworkManager.log

As we can see first my secondary IP is the primary (which seems also wrong), but it changes once I restart NM.

attaching the log

An additional datapoint: if I swap the two ip addresses, meaning making 2a02:168:f00d:caab:0:1:0:2 to IPV6ADDR, it is the chosen primary from the beginning on and it also stays the one when restarting NM.

But I would anyway expect the address in IPV6ADDR to be the primary and not the last one on IPV6ADDR_SECONDARIES (the name suggests something different). Also I would expect that the current primary one stays being the the one when restarting NetworkManager, given there gave been no config changes...

But it seems that there are 2 different ordering mechanisms in place. One at boot time and one when restarting NetworkManager.

Thanks for the logs!!


The address order is wrong during the first activation.

Easy to reproduce with

  DEVICE=eth0
  nmcli connection add type ethernet con-name a autoconnect no ifname "$DEVICE" ipv4.method disabled ipv6.method manual ipv6.addresses "2a02:168:f00d:caab:0:1:0:2/64, 2a02:168:f00d:caab::2/64"
  nmcli connection up a
  ip -6 addr show dev "$DEVICE"


Curiously, after restarting, NetworkManager gets the order right. But other than that, the restart is not relevant here.
After another `nmcli connection up a` it's wrong again.

Btw, restarting NetworkManager is usually the wrong thing to do. Don't do that, unless you have good reasons (there are few good reasons).

> Btw, restarting NetworkManager is usually the wrong thing to do. Don't do
> that, unless you have good reasons (there are few good reasons).

Unrelated, but for the record: NetworkManager got restarted as the rpm got updated and then another script kicked in, which restarted the service due to libs of a running process vanished/got updated and the service was detected responsible. This is when services started connecting with another ip and thus the ip change was detected. If a reboot is more appropriate in that case, this can be be adapted, but as mentioned unrelated and more as background info.

(In reply to Marcel Härri from comment #6)
> > Btw, restarting NetworkManager is usually the wrong thing to do. Don't do
> > that, unless you have good reasons (there are few good reasons).
> 
> Unrelated, but for the record: NetworkManager got restarted as the rpm got
> updated and then another script kicked in, which restarted the service due
> to libs of a running process vanished/got updated and the service was
> detected responsible. This is when services started connecting with another
> ip and thus the ip change was detected. If a reboot is more appropriate in
> that case, this can be be adapted, but as mentioned unrelated and more as
> background info.

thanks for clarifying.

Package update is actually one of the few good reasons to restart NeworkManager!

I am so confused by this bug...



Address selection in kernel
===========================

Kernel selects IPv6 addresses according to https://access.redhat.com/solutions/189153

In case there is a tie, it chooses the address added *last*.
Which means, it's the *first* in `ip -6 addr show` output.

(of course, for IPv4 that's entirely different).



Initscripts
===========

Initscripts comment:

 #  IPV6ADDR=<IPv6 address>[/<prefix length>]: specify primary static IPv6 address
 #  IPV6ADDR_SECONDARIES="<IPv6 address>[/<prefix length>] ..." (optional)

but then ifup-ipv6 does:

# Setup IPv6 address on specified interface
if [ -n "$IPV6ADDR" ]; then
    ipv6_add_addr_on_device $DEVICE $IPV6ADDR || exit 1
fi
...
# Setup additional IPv6 addresses from list, if given
if [ -n "$IPV6ADDR_SECONDARIES" ]; then
    for ipv6addr in $IPV6ADDR_SECONDARIES; do
        ipv6_add_addr_on_device $DEVICE $ipv6addr
    done
fi

meaning, it will add them in the order they appear, and -- see previous point -- the
address added last will be preferred.

Hence, the last of the IPV6ADDR_SECONDARIES will be used.


NetworkManager
==============

For compatibility with initscripts, it also inverses the order.
If you a list of addresses with

  nmcli -f ipv6.addresses connection show "$PROFILE"

then the *last* address should be the primary one. Which is of course unintuitive *sigh*.



Summary1
=========

In comment 0 we see:

  IPV6ADDR=2a02:168:f00d:caab::2/64
  IPV6ADDR_SECONDARIES="2a02:168:f00d:caab::99/64"

and:

2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    [...]
    inet6 2a02:168:f00d:caab::99/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 2a02:168:f00d:caab::2/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:febe:fe88/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
# curl -6 icanhazip.com
2a02:168:f00d:caab::99


so, this part works as intended, although unexpected.



What does not work, is that during a `systemctl restart NetworkManager`, the order gets messed up.
That's the bug...

Yes, I am working around the behavior so far by adding the expected primary address as the last of the secondaries (which is how it used to work since forever) in the kickstart provisioning templates. Which works but is not intuitive.

BUT now when NetworkManager gets restarted it messes them up, which is why I called it inconsistent a behavior. Changing the behavior of the initscripts to the intuitive one (that IPV6ADDR is the primary one) would probably be way too much of a change for RHEL 8 (and 9)...

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: NetworkManager security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4361