Bug 1462260

Summary: cannot assign ipv6 address to a device that has ipv4 address configured
Product: Red Hat Enterprise Linux 7 Reporter: Jan Tluka <jtluka>
Component: NetworkManagerAssignee: Francesco Giudici <fgiudici>
Status: CLOSED ERRATA QA Contact: Desktop QE <desktop-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 7.4CC: aloughla, atragler, bgalvani, fgiudici, lrintel, rkhan, sukulkar, thaller, vbenes
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 13:27:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1470965    
Attachments:
Description Flags
nm log original reproducer
none
nm log without ipv4 address
none
nm log with additional sleep
none
Avoid touching IPv6 on "external" connections without IPv6 configuration.
none
Avoid touching IPv6 on "external" connections without IPv6 configuration. (II) none

Description Jan Tluka 2017-06-16 14:26:42 UTC
Description of problem:

When I run following script after fresh boot (or reload the kernel module for the interface) the IPv6 address is not assigned to the interface. When I turn off NetworkManager the address is assigned without issue.

The script worked fine with older 7.4 version NetworkManager-1.8.0-0.4.rc3.el7.x86_64

I tried with RHEL-7.3 version of NM, NetworkManager-1.4.0-12.el7.x86_64 and it shows the same symptom, so this is not a regression from RHEL-7.3.

<script>

# this can be commented out if the script is run after fresh boot
modprobe -r i40e
modprobe i40e

sleep 5

ip addr flush em1
ethtool -A em1 rx off tx off
ip link set em1 up
# commenting out the line below makes ip6 address assignment successful
ip addr add 192.168.100.2/24 dev em1
# commenting out the line below also makes ipv6 address assignment is successful
# even when the ipv4 is assigned
#sleep 5
ip addr add fe01::1/64 dev em1

sleep 5
ip a show dev em1

</script>

When the script completes I see these addresses on the interface:

172: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid f8bc1213a6e0 state UP qlen 1000
    link/ether f8:bc:12:13:a6:e0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.2/24 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::fabc:12ff:fe13:a6e0/64 scope link 
       valid_lft forever preferred_lft forever


When I run the same script with older NetworkManager-1.8.0-0.4.rc3.el7.x86_64:

176: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid f8bc1213a6e0 state UP qlen 1000
    link/ether f8:bc:12:13:a6:e0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.2/24 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe01::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::fabc:12ff:fe13:a6e0/64 scope link 
       valid_lft forever preferred_lft forever

Version-Release number of selected component (if applicable):
NetworkManager-1.8.0-6.el7.x86_64
kernel-3.10.0-681.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
run the script in description and check the addresses on the interface

Actual results:
IP6 address is not assigned

Expected results:
IP6 address is assigned

Additional info:

NM logs with level=DEBUG attached.

Comment 1 Jan Tluka 2017-06-16 14:30:19 UTC
Tried with latest 7.4 NM, NetworkManager-1.8.0-9.el7.x86_64. It has the same symptom.

Comment 2 Jan Tluka 2017-06-16 14:38:59 UTC
Created attachment 1288383 [details]
nm log original reproducer

NM log when running unmodified reproducer (nothing commented out).

Comment 3 Jan Tluka 2017-06-16 14:40:00 UTC
Created attachment 1288384 [details]
nm log without ipv4 address

NM log when running unmodified reproducer (ip4 address assignment commented out).

Comment 4 Jan Tluka 2017-06-16 14:41:43 UTC
Created attachment 1288385 [details]
nm log with additional sleep

NM log when running modified reproducer (sleep not commented out).

Comment 5 Jan Tluka 2017-06-16 14:42:54 UTC
(In reply to Jan Tluka from comment #3)
> Created attachment 1288384 [details]
> nm log without ipv4 address
> 
> NM log when running unmodified reproducer (ip4 address assignment commented
> out).

Should say __modified__ instead of unmodified. Sorry for noise.

Comment 6 Francesco Giudici 2017-09-20 09:48:48 UTC
Created attachment 1328355 [details]
Avoid touching IPv6 on "external" connections without IPv6 configuration.

When a device managed by NetworkManager is configured manually (adding ip addresses), NetworkManager will track the device configuration with an in-memory only config, marking the device as "external".
Devices marked external should be just tracked but left untouched.
This does not happens here: looking at the logs, what happens is that once first the ipv4 address is added, NM generates the in-memory connection, marking the ipv6.method as "ignore" (the script configures the IPv6 address in the following step).

During in-memory connection activation, NM will process the IPv6 "ignore" method: this implies leaving the IPv6LL address generation to the kernel. To trigger this NM will disable/enable IPv6 on the interface.
This cause a race during the assignment of the IPv6 address.

This NetworkManager behavior looks wrong to me: it should do nothing to IPv6 when method is "ignore" and connection is marked as "external": attached patch fixes this behavior.
Note that if/once IPv6 address is added, IPv6 method is changed to "manual" and a link local address is generated if needed.

Please review the patch, thanks.

Comment 7 Thomas Haller 2017-09-20 12:40:55 UTC
(In reply to Francesco Giudici from comment #6)
> Created attachment 1328355 [details]
> Avoid touching IPv6 on "external" connections without IPv6 configuration.
> 
> When a device managed by NetworkManager is configured manually (adding ip
> addresses), NetworkManager will track the device configuration with an
> in-memory only config, marking the device as "external".
> Devices marked external should be just tracked but left untouched.
> This does not happens here: looking at the logs, what happens is that once
> first the ipv4 address is added, NM generates the in-memory connection,
> marking the ipv6.method as "ignore" (the script configures the IPv6 address
> in the following step).
> 
> During in-memory connection activation, NM will process the IPv6 "ignore"
> method: this implies leaving the IPv6LL address generation to the kernel. To
> trigger this NM will disable/enable IPv6 on the interface.
> This cause a race during the assignment of the IPv6 address.
> 
> This NetworkManager behavior looks wrong to me: it should do nothing to IPv6
> when method is "ignore" and connection is marked as "external": attached
> patch fixes this behavior.
> Note that if/once IPv6 address is added, IPv6 method is changed to "manual"
> and a link local address is generated if needed.
> 
> Please review the patch, thanks.

The description here is insightful. I think it would be good to improve the commit message to contain parts of this text. The comment in bugzilla is rather detached from the code.

In general, I agree, we should not touch nm_device_sys_iface_state_is_external() devices *at all*. Step by step we should fix all the places where we do.

I find the extra parentheses around the condition unnecessary, would however prefer wrapping the line as we do usually, with "&&" at the start.


Anyway, patch lgtm.

Comment 8 Francesco Giudici 2017-09-20 14:06:06 UTC
Created attachment 1328451 [details]
Avoid touching IPv6 on "external" connections without IPv6 configuration. (II)

Comment 9 Francesco Giudici 2017-09-20 14:07:28 UTC
(In reply to Thomas Haller from comment #7)
> (In reply to Francesco Giudici from comment #6)
> > Created attachment 1328355 [details]
> > Avoid touching IPv6 on "external" connections without IPv6 configuration.
> > 
> > When a device managed by NetworkManager is configured manually (adding ip
> > addresses), NetworkManager will track the device configuration with an
> > in-memory only config, marking the device as "external".
> > Devices marked external should be just tracked but left untouched.
> > This does not happens here: looking at the logs, what happens is that once
> > first the ipv4 address is added, NM generates the in-memory connection,
> > marking the ipv6.method as "ignore" (the script configures the IPv6 address
> > in the following step).
> > 
> > During in-memory connection activation, NM will process the IPv6 "ignore"
> > method: this implies leaving the IPv6LL address generation to the kernel. To
> > trigger this NM will disable/enable IPv6 on the interface.
> > This cause a race during the assignment of the IPv6 address.
> > 
> > This NetworkManager behavior looks wrong to me: it should do nothing to IPv6
> > when method is "ignore" and connection is marked as "external": attached
> > patch fixes this behavior.
> > Note that if/once IPv6 address is added, IPv6 method is changed to "manual"
> > and a link local address is generated if needed.
> > 
> > Please review the patch, thanks.
> 
> The description here is insightful. I think it would be good to improve the
> commit message to contain parts of this text. The comment in bugzilla is
> rather detached from the code.
> 
> In general, I agree, we should not touch
> nm_device_sys_iface_state_is_external() devices *at all*. Step by step we
> should fix all the places where we do.
> 
> I find the extra parentheses around the condition unnecessary, would however
> prefer wrapping the line as we do usually, with "&&" at the start.
> 
> 
> Anyway, patch lgtm.

Thanks, updated both the comment and the format of the condition.

Comment 14 errata-xmlrpc 2018-04-10 13:27:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0778