RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1908302 - interface looses link-local address when DHCPv6 fails
Summary: interface looses link-local address when DHCPv6 fails
Keywords:
Status: CLOSED DUPLICATE of bug 1900038
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: NetworkManager
Version: 8.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.4
Assignee: NetworkManager Development Team
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-16 11:27 UTC by Derek Higgins
Modified: 2021-11-03 13:31 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-03 13:31:06 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
journal output associated with comment 2 (117.61 KB, text/plain)
2021-01-08 12:50 UTC, Derek Higgins 		no flags Details
View All

Description Derek Higgins 2020-12-16 11:27:18 UTC 
Description of problem:
In OSP baremetal IPI we are using DHCPv6 to assign IP addresses to the provisioning interface, the DHCP server is initialy running on the bootstrap node, when the
dhcp server goes away (along with the bootstrap node), the interface eventually looses its IP address, this is expected as its not getting DHCPv6 any longer.
The problem is that the node also looses its link local-address which I would have expected to be maintained.

This is effecting dnsmasq in the metal3 pod for provisioning(see https://bugzilla.redhat.com/show_bug.cgi?id=1901040), I've eliminated most of that pod in the setup described below

Version-Release number of selected component (if applicable):
Booted osImageURL:  (47.83.202012030221-0)
Pivoting to: 47.83.202012140447-0 (0b287d5e23513c01269c83cb0b0763229da6d9131b14f6d00e3437d981d826ff)


How reproducible:
Every time when using IPv6 Baremetal IPI 

Steps to Reproduce:
=== Initial setup 
> ip -o link | grep enp
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000\    link/ether 00:69:65:81:4f:2d brd ff:ff:ff:ff:ff:ff
3: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP mode DEFAULT group default qlen 1000\    link/ether 00:69:65:81:4f:2f brd ff:ff:ff:ff:ff:ff

> ip -o addr | grep enp
2: enp1s0    inet6 fd00:1101::5f92:4559:69b8:4d9e/128 scope global dynamic noprefixroute \       valid_lft 3453sec preferred_lft 3453sec
2: enp1s0    inet6 fe80::269:65ff:fe81:4f2d/64 scope link noprefixroute \       valid_lft forever preferred_lft forever

> nmcli con show 'Wired Connection'
connection.id:                          Wired Connection
connection.uuid:                        3b8d52af-9fd1-488c-9ac5-41a38ab9f42a
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              --
connection.autoconnect:                 yes
connection.autoconnect-priority:        0
connection.autoconnect-retries:         -1 (default)
connection.multi-connect:               3 (multiple)
connection.auth-retries:                -1
connection.timestamp:                   1608112431
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
connection.llmnr:                       -1 (default)
connection.wait-device-timeout:         60000
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          no
802-3-ethernet.mac-address:             --
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist:   --
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        --
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            --
802-3-ethernet.wake-on-lan:             default
802-3-ethernet.wake-on-lan-password:    --
ipv4.method:                            auto
ipv4.dns:                               --
ipv4.dns-search:                        --
ipv4.dns-options:                       --
ipv4.dns-priority:                      0
ipv4.addresses:                         --
ipv4.gateway:                           --
ipv4.routes:                            --
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.routing-rules:                     --
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-iaid:                         --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.dhcp-fqdn:                         --
ipv4.dhcp-hostname-flags:               0x0 (none)
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv4.dhcp-vendor-class-identifier:      --
ipv6.method:                            auto
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       --
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.routing-rules:                     --
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     eui64
ipv6.ra-timeout:                        0 (default)
ipv6.dhcp-duid:                         --
ipv6.dhcp-iaid:                         --
ipv6.dhcp-timeout:                      0 (default)
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.dhcp-hostname-flags:               0x0 (none)
ipv6.token:                             --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           Wired Connection
GENERAL.UUID:                           3b8d52af-9fd1-488c-9ac5-41a38ab9f42a
GENERAL.DEVICES:                        enp1s0
GENERAL.IP-IFACE:                       enp1s0
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        no
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/1
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP6.ADDRESS[1]:                         fd00:1101::5f92:4559:69b8:4d9e/128
IP6.ADDRESS[2]:                         fe80::269:65ff:fe81:4f2d/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fd00:1101::/64, nh = ::, mt = 100
IP6.ROUTE[2]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[3]:                           dst = fe80::/64, nh = ::, mt = 100
IP6.ROUTE[4]:                           dst = fd00:1101::5f92:4559:69b8:4d9e/128, nh = ::, mt = 100
DHCP6.OPTION[1]:                        ip6_address = fd00:1101::5f92:4559:69b8:4d9e

# Should this be "0" for eui64(confirmed by nmcli above)?
# ipv6.addr-gen-mode:                     eui64
> cat /proc/sys/net/ipv6/conf/enp1s0/addr_gen_mode 
1


=== Eventually DHCPv6 fails


Dec 16 10:50:15 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115815.7174] dhcp6 (enp1s0): state changed bound -> fail
Dec 16 10:50:15 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115815.7178] device (enp1s0): DHCPv6: trying to acquire a new lease within 90 seconds
...
Dec 16 10:51:45 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115905.5437] device (enp1s0): DHCPv6: grace period expired
Dec 16 10:51:45 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115905.5441] device (enp1s0): state change: activated -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
Dec 16 10:51:45 master-2.ostest.test.metalkube.org NetworkManager[1405]: <warn>  [1608115905.5458] device (enp1s0): Activation: failed for connection 'Wired Connection'
Dec 16 10:51:45 master-2.ostest.test.metalkube.org dbus-daemon[1216]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=1405 comm="/usr/sbin/NetworkManager --no-daemon " label="system_u:system_r:NetworkManager_t:s0")
Dec 16 10:51:45 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115905.5471] device (enp1s0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Dec 16 10:51:45 master-2.ostest.test.metalkube.org ovs-vswitchd[1387]: ovs|00042|bridge|ERR|interface br-ex: ignoring mac in Interface record (use Bridge record to set local port's mac)
Dec 16 10:51:45 master-2.ostest.test.metalkube.org systemd[1]: Starting Network Manager Script Dispatcher Service...
Dec 16 10:51:45 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115905.5588] dhcp4 (enp1s0): canceled DHCP transaction
Dec 16 10:51:45 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115905.5589] dhcp4 (enp1s0): state changed timeout -> done
Dec 16 10:51:45 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115905.5598] dhcp6 (enp1s0): canceled DHCP transaction
Dec 16 10:51:45 master-2.ostest.test.metalkube.org NetworkManager[1405]: <info>  [1608115905.5598] dhcp6 (enp1s0): state changed fail -> done
Dec 16 10:51:45 master-2.ostest.test.metalkube.org dbus-daemon[1216]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Dec 16 10:51:45 master-2.ostest.test.metalkube.org systemd[1]: Started Network Manager Script Dispatcher Service.
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: + [[ OVNKubernetes == \O\V\N\K\u\b\e\r\n\e\t\e\s ]]
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: + [[ Wired Connection == \W\i\r\e\d\ \C\o\n\n\e\c\t\i\o\n ]]
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: + echo 'Refusing to modify default connection.'
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: Refusing to modify default connection.
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: + exit 0
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: + '[' -z ']'
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: + echo 'Not a DHCP6 address. Ignoring.'
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: Not a DHCP6 address. Ignoring.
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: + exit 0
Dec 16 10:51:45 master-2.ostest.test.metalkube.org root[117565]: NM mdns-hostname triggered by down.
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: <13>Dec 16 10:51:45 root: NM mdns-hostname triggered by down.
Dec 16 10:51:45 master-2.ostest.test.metalkube.org root[117568]: Hostname changed: master-2.ostest.test.metalkube.org
Dec 16 10:51:45 master-2.ostest.test.metalkube.org nm-dispatcher[117553]: <13>Dec 16 10:51:45 root: Hostname changed: master-2.ostest.test.metalkube.org
Dec 16 10:51:45 master-2.ostest.test.metalkube.org systemd[1]: Starting Generate console-login-helper-messages issue snippet...
Dec 16 10:51:46 master-2.ostest.test.metalkube.org systemd[1]: console-login-helper-messages-issuegen.service: Succeeded.
Dec 16 10:51:46 master-2.ostest.test.metalkube.org systemd[1]: Started Generate console-login-helper-messages issue snippet.
Dec 16 10:51:46 master-2.ostest.test.metalkube.org systemd[1]: console-login-helper-messages-issuegen.service: Consumed 15ms CPU time
Dec 16 10:51:47 master-2.ostest.test.metalkube.org sshd[117616]: Accepted publickey for core from fd2e:6f44:5dd8:c956::1 port 47526 ssh2: RSA SHA256:6I+mKGCpLPuPSipvS+yyTEfQu9P0y59cm+1gfcAEkkc
Dec 16 10:51:47 master-2.ostest.test.metalkube.org systemd-logind[1310]: New session 226 of user core.
Dec 16 10:51:47 master-2.ostest.test.metalkube.org systemd[1]: Started Session 226 of user core.
Dec 16 10:51:47 master-2.ostest.test.metalkube.org sshd[117616]: pam_unix(sshd:session): session opened for user core by (uid=0)
Dec 16 10:51:47 master-2.ostest.test.metalkube.org sshd[117619]: Received disconnect from fd2e:6f44:5dd8:c956::1 port 47526:11: disconnected by user
Dec 16 10:51:47 master-2.ostest.test.metalkube.org sshd[117619]: Disconnected from user core fd2e:6f44:5dd8:c956::1 port 47526
Dec 16 10:51:47 master-2.ostest.test.metalkube.org sshd[117616]: pam_unix(sshd:session): session closed for user core
Dec 16 10:51:47 master-2.ostest.test.metalkube.org systemd-logind[1310]: Session 226 logged out. Waiting for processes to exit.
Dec 16 10:51:47 master-2.ostest.test.metalkube.org systemd[1]: session-226.scope: Succeeded.
Dec 16 10:51:47 master-2.ostest.test.metalkube.org systemd[1]: session-226.scope: Consumed 44ms CPU time
Dec 16 10:51:47 master-2.ostest.test.metalkube.org systemd-logind[1310]: Removed session 226.


=== The address is gone
[root@master-2 core]# ip -o addr | grep enp

=== but I can get it back by writing to addr_gen_mode
[root@master-2 core]# echo 0 > /proc/sys/net/ipv6/conf/enp1s0/addr_gen_mode 
[root@master-2 core]# ip -o addr | grep enp
2: enp1s0    inet6 fe80::269:65ff:fe81:4f2d/64 scope link \       valid_lft forever preferred_lft forever
Comment 1 Micah Abbott 2020-12-17 16:01:29 UTC 
We'll need to investigate this further, but I would be surprised if this behavior is limited to just RHCOS, as we are using the same NetworkManager that is available in RHEL8.

It would be useful if it you could try to reduce this problem to its most simple reproducer and see if it reproduces on a vanilla RHEL 8.3 system with the same version of NetworkManger (NetworkManager-1.26.0-9.el8_3) as found in RHCOS 47.83.202012030221-0
Comment 2 Derek Higgins 2021-01-08 12:46:26 UTC 
(In reply to Micah Abbott from comment #1)
> We'll need to investigate this further, but I would be surprised if this
> behavior is limited to just RHCOS, as we are using the same NetworkManager
> that is available in RHEL8.
> 
> It would be useful if it you could try to reduce this problem to its most
> simple reproducer and see if it reproduces on a vanilla RHEL 8.3 system with
> the same version of NetworkManger (NetworkManager-1.26.0-9.el8_3) as found
> in RHCOS 47.83.202012030221-0

Yup, the same thing happens on RHEL 8.3 (although this time ipv6.addr-gen-mode is set to stable-privacy)

[root@h2 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.3 (Ootpa)

[root@h2 ~]# rpm -qa | grep -i networkma
NetworkManager-libnm-1.26.0-12.el8_3.x86_64
NetworkManager-team-1.26.0-12.el8_3.x86_64
NetworkManager-1.26.0-12.el8_3.x86_64
NetworkManager-tui-1.26.0-12.el8_3.x86_64

[root@h2 ~]# ip -o addr show dev enp7s0
3: enp7s0    inet6 fd00:1101::64b7/128 scope global dynamic noprefixroute \       valid_lft 550sec preferred_lft 550sec
3: enp7s0    inet6 fe80::2a7:ad19:3892:6777/64 scope link noprefixroute \       valid_lft forever preferred_lft forever


[root@h2 ~]# cat /proc/sys/net/ipv6/conf/enp7s0/addr_gen_mode 
1

[root@h2 ~]# nmcli con show ethernet | grep -i ipv6.addr-gen-mode
ipv6.addr-gen-mode:                     stable-privacy

# After killing the dhcpserver (which was also sending out the router adverts)
# Both ipaddresses are lost, the link local address eventually comes back for a few minutes
# and is then lost again 
[root@h2 ~]# while true ; do sleep 10 ; date ; ip -o addr show dev enp7s0 ; done
Fri  8 Jan 07:21:58 EST 2021                                                                                                                                                                                      
3: enp7s0    inet6 fd00:1101::64b7/128 scope global dynamic noprefixroute \       valid_lft 14sec preferred_lft 14sec                                                                                             
3: enp7s0    inet6 fe80::2a7:ad19:3892:6777/64 scope link noprefixroute \       valid_lft forever preferred_lft forever                                                                                           
Fri  8 Jan 07:22:08 EST 2021                                                                                                                                                                                      
3: enp7s0    inet6 fd00:1101::64b7/128 scope global dynamic noprefixroute \       valid_lft 4sec preferred_lft 4sec    
3: enp7s0    inet6 fe80::2a7:ad19:3892:6777/64 scope link noprefixroute \       valid_lft forever preferred_lft forever                                                                                           
Fri  8 Jan 07:22:18 EST 2021                                                                                           
3: enp7s0    inet6 fe80::2a7:ad19:3892:6777/64 scope link noprefixroute \       valid_lft forever preferred_lft forever                                                                                           
...
Fri  8 Jan 07:25:58 EST 2021                                                                                                                                                                                      
3: enp7s0    inet6 fe80::2a7:ad19:3892:6777/64 scope link noprefixroute \       valid_lft forever preferred_lft forever                                                                                           
Fri  8 Jan 07:26:08 EST 2021                                                                                                                                                                                      
...
Fri  8 Jan 07:30:58 EST 2021                                                                                         
Fri  8 Jan 07:31:08 EST 2021                                                                                                                                                                                      
3: enp7s0    inet6 fe80::2a7:ad19:3892:6777/64 scope link noprefixroute \       valid_lft forever preferred_lft forever                                                                                           
...
Fri  8 Jan 07:33:52 EST 2021                                                                                           
3: enp7s0    inet6 fe80::2a7:ad19:3892:6777/64 scope link noprefixroute \       valid_lft forever preferred_lft forever                                                                                           
Fri  8 Jan 07:34:02 EST 2021                                                                                           
Fri  8 Jan 07:34:12 EST 2021                                                                                           
Fri  8 Jan 07:34:22 EST 2021
Comment 3 Derek Higgins 2021-01-08 12:50:09 UTC 
Created attachment 1745581 [details]
journal output associated with comment 2
Comment 4 Micah Abbott 2021-01-08 13:54:45 UTC 
Thanks for the additional data, Derek.  Based on your findings, I am going to send this over to the NetworkManager team for additional action.
Comment 8 Beniamino Galvani 2021-07-09 08:18:39 UTC 
When the DHCPv6 server goes away, the connection fails and the device is disconnected removing IPv6 addresses.

To avoid that the connection fails, you can set ipv6.dhcp-timeout to infinity. From kernel command line this can be done by adding option 'rd.net.timeout.dhcp=infinity'.
Comment 9 Till Maas 2021-07-09 09:16:39 UTC 
Derek, please confirm that using the infinity timeout solves the issue for you. Thanks.
Comment 10 Gris Ge 2021-09-18 07:19:26 UTC 
Hi Derek,

Please try commands which should fix your problem.

nmcli connection modify <connection_name> ipv6.ra_timeout infinity ipv6.dhcp-timeout infinity


We are also working in bug 1900038 to improve the user experience on DHCP timeout.

Thank you!
Comment 11 Gris Ge 2021-11-03 13:31:06 UTC 
Closing as duplicate of bug 1900038 .

Please feel free to reopen this bug if you think other help is required.

*** This bug has been marked as a duplicate of bug 1900038 ***
Note You need to log in before you can comment on or make changes to this bug.