Bug 1548237 - network interface ipv6 address missed after down and up it while NetworkManger service is running
Summary: network interface ipv6 address missed after down and up it while NetworkMange...
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: NetworkManager
Version: 7.5
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Beniamino Galvani
QA Contact: Desktop QE
URL:
Whiteboard:
Keywords:
: 1601669 (view as bug list)
Depends On:
Blocks: 1654714
TreeView+ depends on / blocked
 
Reported: 2018-02-23 02:27 UTC by Xiao, Liang
Modified: 2019-05-28 06:59 UTC (History)
15 users (show)

(edit)
Clone Of:
(edit)
Last Closed:


Attachments (Terms of Use)

Description Xiao, Liang 2018-02-23 02:27:04 UTC
Created a RHEL7.5 hvm guest residing Fedora27 xen4 host. NetworkManager service was started by default.
network interface's ipv6 address missed after down and up it.

Here is log;
# ip address show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:13:24:36:45:57 brd ff:ff:ff:ff:ff:ff
    inet 10.66.0.124/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0
       valid_lft 86263sec preferred_lft 86263sec
    inet6 fe80::1213:24ff:fe36:4557/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
# ip link set dev eth0 down
# ip address show dev eth0
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 10:13:24:36:45:57 brd ff:ff:ff:ff:ff:ff
    inet 10.66.0.124/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0
       valid_lft 86228sec preferred_lft 86228sec
# ip link set dev eth0 up
- No ipv6 address after interface up again.
# ip address show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:13:24:36:45:57 brd ff:ff:ff:ff:ff:ff
    inet 10.66.0.124/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0
       valid_lft 86398sec preferred_lft 86398sec

Version-Release number of selected component (if applicable):
Host: Fedora27 xen-4.9.1

RHEL Version:
RHEL7.5(3.10.0-855.el7.x86_64)

How reproducible:
100%

Steps to Reproduce:
1. Create a hvm guest residing on Fedora27 xen4 host.
2. down and up eth0 in guest.
#ip link set dev eth0 down
#ip link set dev eth0 up
3. Check ipv6 address of eth0
# ip address show dev eth0

Actual results:
No ipv6 address found

Expected results:
Interface up with ipv6 assigned.
Additional info:
- Cannot reproduce it on Fedora27 and Ubuntu.
- Cannot reproduce it if only enable network.service
- Can reproduce this bug with RHEL7.4 GA release installed, so it is not a regression.
- Only can reproduce it if enable NetworkManager service
- Can reproduce it in bare metal system(tg3 nic driver), so it is not a xen related bug
- Can reproduce it both using "ip" or "ifconfig" to down and up network interface.

Comment 2 Thomas Haller 2018-02-23 20:08:15 UTC
If you set a interface down (ip link set down $NAME), then kernel will remove all IPv6 addresses.

NetworkManager configures all links with "addrgenmode none". This tells kernel not to automatically add IPv6 link local addresses. Link local addresses are instead added by NetworkManager. This is necessary because:

  - NetworkManager always sets links IFF_UP, even if the link is logically 
    disconnected. It does so, to get carrier-changed events from kernel. But if 
    the link is logically disconnected, it must have no IP addresses.

  - support private stable IPv6 link local addresses (RFC7217).
    See addr-gen-mode in `man nm-settings`.

You can see that with

  ip -d link show



If you don't use NetworkManager, the link commonly has "addgenmode eui64". In that case, setting a link down will remove IPv6 addresses but bringing it up again will prompt kernel to re-add the IPv6 link local address. If you re-up a link with "addrgenmode none", nobody is re-adding the IPv6 link local address.


Not even NetworkManager will re-add the IPv6 address. In general, it's probably a bad idea to mess with interfaces that are managed by NetworkManager. But if you do so, NetworkManager tries hard not to interfere with your external changes. In this case, you set the link down and kernel removed the IPv6 addresses. NetworkManager sees that somebody did something, but it won't revert the external changes (how would it know to do so?).

You might wanna call `nmcli device reapply "$NAME`, which basically tells NetworkManager to restore all previously configured IP addresses. Or go through a full down+up cycle with `nmcli connection up "$CON"`.




> Additional info:
> - Cannot reproduce it on Fedora27 and Ubuntu.

Which versions of NetworkManager specifically? That sounds like a bug. It also doesn't sound very plausible, because Fedora27 and RHEL-7.4 both ship a rather similar NetworkManager (1.8.x) -- and below you said, that RHEL-7.4 shows the issue.


> - Cannot reproduce it if only enable network.service

In that case, the link likely is "addrgenmode eui64". Note that this will only restore IPv6 link local addresses. All other IPv6 addresses are gone -- arguably, kernel (depending on sysctl configuration) might do autoconf again and receive the same prefixes again. But manually added addresses are lost either way.

> - Can reproduce this bug with RHEL7.4 GA release installed, so it is not a
>   regression.

As said, unclear why Fedora 27 would behave differently then rhel-7.4.


> - Only can reproduce it if enable NetworkManager service

As explained, that's because NetworkManager sets "addrgenmode none".

> - Can reproduce it in bare metal system(tg3 nic driver), so it is not a xen 
>   related bug
> - Can reproduce it both using "ip" or "ifconfig" to down and up network 
>   interface.

Both is not surprising.



I would argue this works as expected. Only thing that surprises me is the claim that Fedora 27 and Ubuntu behave differently. That would need some investigation/clarification.



Can you explain why you are setting the interface down in the first place? What are you trying to achive? Thanks



Sidenote: often logfiles of NetworkManager with level=TRACE are helpful. See hints about logging at https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf

Comment 3 Xiao, Liang 2018-02-24 08:21:45 UTC
(In reply to Thomas Haller from comment #2)

Thanks for your detail explanation. So, the behavior under RHEL7.5 is expected as "addrgenmode none". 

> > Additional info:
> > - Cannot reproduce it on Fedora27 and Ubuntu.
> 
> Which versions of NetworkManager specifically? That sounds like a bug. It
> also doesn't sound very plausible, because Fedora27 and RHEL-7.4 both ship a
> rather similar NetworkManager (1.8.x) -- and below you said, that RHEL-7.4
> shows the issue.
Below is the NetworkManager version in Fedora27, RHEL7.5 and RHEL8 nightly build.
NetworkManager-1.8.2-3.fc27.2.x86_64
NetworkManager-1.10.2-13.el7.x86_64
NetworkManager-1.10.4-1.el8+5.x86_64

> In that case, the link likely is "addrgenmode eui64". Note that this will
> only restore IPv6 link local addresses. All other IPv6 addresses are gone --
> arguably, kernel (depending on sysctl configuration) might do autoconf again
> and receive the same prefixes again. But manually added addresses are lost
> either way.
Exactly same as you said.

> I would argue this works as expected. Only thing that surprises me is the
> claim that Fedora 27 and Ubuntu behave differently. That would need some
> investigation/clarification.
I agree that it works as expected after tried. As ubuntu setting is "addrgenmode eui64", I will not compare with it now. 
I double checked it on Fedora27 and RHEL8. As RHEL8 nightly build also behave differently. Could you help confirm whether I need to file a bug to RHEL8 project? Thanks
Below are my steps:

[root@dhcp-0-105 ~]# ip -d link show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode none numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_max_segs 65535 
[root@dhcp-0-105 ~]# ip a show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff
    inet 10.66.0.105/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0
       valid_lft 86332sec preferred_lft 86332sec
    inet6 fe80::b564:8ca0:69c1:3479/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@dhcp-0-105 ~]# ip link set eth0 down
[root@dhcp-0-105 ~]# ip a show dev eth0
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff
    inet 10.66.0.105/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0
       valid_lft 86325sec preferred_lft 86325sec
[root@dhcp-0-105 ~]# ip link set eth0 up
[root@dhcp-0-105 ~]# ip a show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff
    inet 10.66.0.105/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0
       valid_lft 86316sec preferred_lft 86316sec
    inet6 fe80::b564:8ca0:69c1:3479/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
> 
> Can you explain why you are setting the interface down in the first place?
> What are you trying to achive? Thanks
I set the link down during the iperf3 load running. To see whether there are any  exceptions happen. IPv4 connection restored after link down and up, but IPv6 did not.
And compared with Fedora27, the behavior is different, so I filed this bug.

Comment 4 sushil kulkarni 2018-06-07 19:37:13 UTC
Thomas,

Is anything expected here w.r.t RHEL 8? Wasn't able to tell based on Xiao's question in c#3.

Thanks!
Sushil

Comment 5 Thomas Haller 2018-06-25 11:43:46 UTC
as discussed, I think we see here expected behaviour, and I would close this as NOTABUG.

Objections anyone?

Comment 6 Xiao, Liang 2018-07-12 14:29:18 UTC
As here is expected behavior in RHEL7, I will file another bug against RHEL8 to track different behavior for tracking. Please feel free to close it. Thanks

Comment 7 Xiao, Liang 2018-07-17 01:38:38 UTC
Filed a new bug 1601669 to track it on RHEL8.

Comment 8 Thomas Haller 2018-07-17 13:57:23 UTC
*** Bug 1601669 has been marked as a duplicate of this bug. ***

Comment 9 Thomas Haller 2018-07-17 14:25:27 UTC
(In reply to Xiao, Liang from comment #6)
> As here is expected behavior in RHEL7, I will file another bug against RHEL8
> to track different behavior for tracking. Please feel free to close it.
> Thanks

Issues for NM are always fixed upstream first. For upstream, we are generally cautious to not break behaviour in a bad way. When we change behaviour, it is usually also done in a way that is suitable for RHEL minor releases too -- there are exceptions to that, where we carry small downstream patches.

Anyway, I am of the opinion, that this is not a bug, but desired behaviour (which is up for debate, of course). However: it is not declared desired behaviour, based on the assumption that it wouldn't be suitable for rhel-7. Hence, cloning for rhel-8 is not necessary to solve this issue, so I closed bug 1601669 as duplicate.

Still need to figure out, what the pros + cons here are, and the way forward. I still lean to NOTABUG.

Thanks. TODO.

Comment 10 Beniamino Galvani 2018-11-27 09:33:34 UTC
I think NM could be improved to better handle this situation. On
master, when the interface goes up again we call
ip_merge_and_apply(commit=1) to reapply the IP configuration, but we
fail to add IPv6 addresses because they were removed from the AC IP
config in update_ext_ip_config(). The ndisc instance keeps running and
re-adds addresses only when the next (unsolicited) RA is received.

Ideally NM should be able to detect that addresses were removed not
because the user did so but because the interface went down, and
should re-add them immediately when the interface goes up.

Comment 11 Thomas Haller 2018-11-27 09:41:32 UTC
related: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=f069c98cc95494891215ddf261661afb742744ba

This solves a similar problem for IP routes...

Comment 12 Beniamino Galvani 2019-03-09 16:40:13 UTC
https://github.com/NetworkManager/NetworkManager/pull/314

Comment 13 Beniamino Galvani 2019-03-12 16:39:57 UTC
Fixed upstream.

Comment 15 Xiao, Liang 2019-05-05 07:26:18 UTC
If the interface only has ipv6 local link address, it cannot be recovered after port down and up. But it works in RHEL8.0. Could you help check the result?

[root@dhcp-14-171 ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff
    inet 10.66.14.171/23 brd 10.66.15.255 scope global dynamic noprefixroute eth0
       valid_lft 85994sec preferred_lft 85994sec
    inet6 fe80::216:3eff:fefb:7835/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@dhcp-14-171 ~]# ip link set dev eth0 down
[root@dhcp-14-171 ~]# ip link set dev eth0 up
[root@dhcp-14-171 ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff
    inet 10.66.14.171/23 brd 10.66.15.255 scope global dynamic noprefixroute eth0
       valid_lft 86399sec preferred_lft 86399sec

If the port has 2 ipv6 addresses, both can be recovered now.

[root@vm-198-147 ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:13:24:35:45:2e brd ff:ff:ff:ff:ff:ff
    inet 10.73.198.147/22 brd 10.73.199.255 scope global noprefixroute dynamic eth0
       valid_lft 42618sec preferred_lft 42618sec
    inet6 2620:52:0:49c4:1213:24ff:fe35:452e/64 scope global noprefixroute dynamic 
       valid_lft 2591948sec preferred_lft 604748sec
    inet6 fe80::1213:24ff:fe35:452e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@vm-198-147 ~]# ip link set dev eth0 down
[root@vm-198-147 ~]# ip link set dev eth0 up
[root@vm-198-147 ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:13:24:35:45:2e brd ff:ff:ff:ff:ff:ff
    inet 10.73.198.147/22 brd 10.73.199.255 scope global noprefixroute dynamic eth0
       valid_lft 43199sec preferred_lft 43199sec
    inet6 2620:52:0:49c4:1213:24ff:fe35:452e/64 scope global noprefixroute dynamic 
       valid_lft 2592000sec preferred_lft 604800sec
    inet6 fe80::1213:24ff:fe35:452e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Comment 16 Beniamino Galvani 2019-05-06 08:32:23 UTC
(In reply to Xiao, Liang from comment #15)
> If the interface only has ipv6 local link address, it cannot be recovered
> after port down and up. But it works in RHEL8.0. Could you help check the
> result?

Which NM version are you using?

Comment 17 Xiao, Liang 2019-05-06 08:55:39 UTC
(In reply to Beniamino Galvani from comment #16)
> (In reply to Xiao, Liang from comment #15)
> > If the interface only has ipv6 local link address, it cannot be recovered
> > after port down and up. But it works in RHEL8.0. Could you help check the
> > result?
> 
> Which NM version are you using?
I am using the default NM in RHEL-7.7-20190502.1 build.

[root@dhcp-14-158 ~]# rpm -qa|grep -i networkm
NetworkManager-libnm-1.18.0-1.el7.x86_64
NetworkManager-team-1.18.0-1.el7.x86_64
NetworkManager-1.18.0-1.el7.x86_64
NetworkManager-config-server-1.18.0-1.el7.noarch
NetworkManager-glib-1.18.0-1.el7.x86_64
NetworkManager-tui-1.18.0-1.el7.x86_64

Comment 18 Vladimir Benes 2019-05-07 10:29:01 UTC
This seems to work well:
12: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:ed:7d:1c brd ff:ff:ff:ff:ff:ff
    inet 10.19.189.159/23 brd 10.19.189.255 scope global noprefixroute dynamic eth10
       valid_lft 86221sec preferred_lft 86221sec
    inet6 2620:52:0:13bc:a800:7f95:46de:3901/64 scope global noprefixroute dynamic 
       valid_lft 2591968sec preferred_lft 604768sec
    inet6 fe80::263:4c54:f831:848c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip link set eth10 down 
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip link set eth10 up
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip a s eth10
12: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:ed:7d:1c brd ff:ff:ff:ff:ff:ff
    inet 10.19.189.159/23 brd 10.19.189.255 scope global noprefixroute dynamic eth10
       valid_lft 86398sec preferred_lft 86398sec
    inet6 2620:52:0:13bc:a800:7f95:46de:3901/64 scope global noprefixroute dynamic 
       valid_lft 2591999sec preferred_lft 604799sec
    inet6 fe80::263:4c54:f831:848c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

but this does not (as above comments said)
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip a s eth9
11: eth9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:a9:a7:7c brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.216/24 brd 192.168.100.255 scope global noprefixroute dynamic eth9
       valid_lft 3599sec preferred_lft 3599sec
    inet6 fe80::470b:30db:ab2c:e918/64 scope link tentative noprefixroute 
       valid_lft forever preferred_lft forever
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip link set dev eth9 down
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip link set dev eth9 up
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip a s eth9
11: eth9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:a9:a7:7c brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.216/24 brd 192.168.100.255 scope global noprefixroute dynamic eth9
       valid_lft 3599sec preferred_lft 3599sec


Note You need to log in before you can comment on or make changes to this bug.