Bug 1462260 - cannot assign ipv6 address to a device that has ipv4 address configured
cannot assign ipv6 address to a device that has ipv4 address configured
Status: ON_QA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: NetworkManager (Show other bugs)
7.4
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Francesco Giudici
Desktop QE
:
Depends On:
Blocks: 1470965
  Show dependency treegraph
 
Reported: 2017-06-16 10:26 EDT by Jan Tluka
Modified: 2017-10-27 08:10 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
nm log original reproducer (94.92 KB, text/x-vhdl)
2017-06-16 10:38 EDT, Jan Tluka
no flags Details
nm log without ipv4 address (89.14 KB, text/x-vhdl)
2017-06-16 10:40 EDT, Jan Tluka
no flags Details
nm log with additional sleep (93.04 KB, text/x-vhdl)
2017-06-16 10:41 EDT, Jan Tluka
no flags Details
Avoid touching IPv6 on "external" connections without IPv6 configuration. (1.01 KB, patch)
2017-09-20 05:48 EDT, Francesco Giudici
no flags Details | Diff
Avoid touching IPv6 on "external" connections without IPv6 configuration. (II) (2.02 KB, patch)
2017-09-20 10:06 EDT, Francesco Giudici
no flags Details | Diff

  None (edit)
Description Jan Tluka 2017-06-16 10:26:42 EDT
Description of problem:

When I run following script after fresh boot (or reload the kernel module for the interface) the IPv6 address is not assigned to the interface. When I turn off NetworkManager the address is assigned without issue.

The script worked fine with older 7.4 version NetworkManager-1.8.0-0.4.rc3.el7.x86_64

I tried with RHEL-7.3 version of NM, NetworkManager-1.4.0-12.el7.x86_64 and it shows the same symptom, so this is not a regression from RHEL-7.3.

<script>

# this can be commented out if the script is run after fresh boot
modprobe -r i40e
modprobe i40e

sleep 5

ip addr flush em1
ethtool -A em1 rx off tx off
ip link set em1 up
# commenting out the line below makes ip6 address assignment successful
ip addr add 192.168.100.2/24 dev em1
# commenting out the line below also makes ipv6 address assignment is successful
# even when the ipv4 is assigned
#sleep 5
ip addr add fe01::1/64 dev em1

sleep 5
ip a show dev em1

</script>

When the script completes I see these addresses on the interface:

172: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid f8bc1213a6e0 state UP qlen 1000
    link/ether f8:bc:12:13:a6:e0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.2/24 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::fabc:12ff:fe13:a6e0/64 scope link 
       valid_lft forever preferred_lft forever


When I run the same script with older NetworkManager-1.8.0-0.4.rc3.el7.x86_64:

176: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid f8bc1213a6e0 state UP qlen 1000
    link/ether f8:bc:12:13:a6:e0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.2/24 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe01::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::fabc:12ff:fe13:a6e0/64 scope link 
       valid_lft forever preferred_lft forever

Version-Release number of selected component (if applicable):
NetworkManager-1.8.0-6.el7.x86_64
kernel-3.10.0-681.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
run the script in description and check the addresses on the interface

Actual results:
IP6 address is not assigned

Expected results:
IP6 address is assigned

Additional info:

NM logs with level=DEBUG attached.
Comment 1 Jan Tluka 2017-06-16 10:30:19 EDT
Tried with latest 7.4 NM, NetworkManager-1.8.0-9.el7.x86_64. It has the same symptom.
Comment 2 Jan Tluka 2017-06-16 10:38 EDT
Created attachment 1288383 [details]
nm log original reproducer

NM log when running unmodified reproducer (nothing commented out).
Comment 3 Jan Tluka 2017-06-16 10:40 EDT
Created attachment 1288384 [details]
nm log without ipv4 address

NM log when running unmodified reproducer (ip4 address assignment commented out).
Comment 4 Jan Tluka 2017-06-16 10:41 EDT
Created attachment 1288385 [details]
nm log with additional sleep

NM log when running modified reproducer (sleep not commented out).
Comment 5 Jan Tluka 2017-06-16 10:42:54 EDT
(In reply to Jan Tluka from comment #3)
> Created attachment 1288384 [details]
> nm log without ipv4 address
> 
> NM log when running unmodified reproducer (ip4 address assignment commented
> out).

Should say __modified__ instead of unmodified. Sorry for noise.
Comment 6 Francesco Giudici 2017-09-20 05:48 EDT
Created attachment 1328355 [details]
Avoid touching IPv6 on "external" connections without IPv6 configuration.

When a device managed by NetworkManager is configured manually (adding ip addresses), NetworkManager will track the device configuration with an in-memory only config, marking the device as "external".
Devices marked external should be just tracked but left untouched.
This does not happens here: looking at the logs, what happens is that once first the ipv4 address is added, NM generates the in-memory connection, marking the ipv6.method as "ignore" (the script configures the IPv6 address in the following step).

During in-memory connection activation, NM will process the IPv6 "ignore" method: this implies leaving the IPv6LL address generation to the kernel. To trigger this NM will disable/enable IPv6 on the interface.
This cause a race during the assignment of the IPv6 address.

This NetworkManager behavior looks wrong to me: it should do nothing to IPv6 when method is "ignore" and connection is marked as "external": attached patch fixes this behavior.
Note that if/once IPv6 address is added, IPv6 method is changed to "manual" and a link local address is generated if needed.

Please review the patch, thanks.
Comment 7 Thomas Haller 2017-09-20 08:40:55 EDT
(In reply to Francesco Giudici from comment #6)
> Created attachment 1328355 [details]
> Avoid touching IPv6 on "external" connections without IPv6 configuration.
> 
> When a device managed by NetworkManager is configured manually (adding ip
> addresses), NetworkManager will track the device configuration with an
> in-memory only config, marking the device as "external".
> Devices marked external should be just tracked but left untouched.
> This does not happens here: looking at the logs, what happens is that once
> first the ipv4 address is added, NM generates the in-memory connection,
> marking the ipv6.method as "ignore" (the script configures the IPv6 address
> in the following step).
> 
> During in-memory connection activation, NM will process the IPv6 "ignore"
> method: this implies leaving the IPv6LL address generation to the kernel. To
> trigger this NM will disable/enable IPv6 on the interface.
> This cause a race during the assignment of the IPv6 address.
> 
> This NetworkManager behavior looks wrong to me: it should do nothing to IPv6
> when method is "ignore" and connection is marked as "external": attached
> patch fixes this behavior.
> Note that if/once IPv6 address is added, IPv6 method is changed to "manual"
> and a link local address is generated if needed.
> 
> Please review the patch, thanks.

The description here is insightful. I think it would be good to improve the commit message to contain parts of this text. The comment in bugzilla is rather detached from the code.

In general, I agree, we should not touch nm_device_sys_iface_state_is_external() devices *at all*. Step by step we should fix all the places where we do.

I find the extra parentheses around the condition unnecessary, would however prefer wrapping the line as we do usually, with "&&" at the start.


Anyway, patch lgtm.
Comment 8 Francesco Giudici 2017-09-20 10:06 EDT
Created attachment 1328451 [details]
Avoid touching IPv6 on "external" connections without IPv6 configuration. (II)
Comment 9 Francesco Giudici 2017-09-20 10:07:28 EDT
(In reply to Thomas Haller from comment #7)
> (In reply to Francesco Giudici from comment #6)
> > Created attachment 1328355 [details]
> > Avoid touching IPv6 on "external" connections without IPv6 configuration.
> > 
> > When a device managed by NetworkManager is configured manually (adding ip
> > addresses), NetworkManager will track the device configuration with an
> > in-memory only config, marking the device as "external".
> > Devices marked external should be just tracked but left untouched.
> > This does not happens here: looking at the logs, what happens is that once
> > first the ipv4 address is added, NM generates the in-memory connection,
> > marking the ipv6.method as "ignore" (the script configures the IPv6 address
> > in the following step).
> > 
> > During in-memory connection activation, NM will process the IPv6 "ignore"
> > method: this implies leaving the IPv6LL address generation to the kernel. To
> > trigger this NM will disable/enable IPv6 on the interface.
> > This cause a race during the assignment of the IPv6 address.
> > 
> > This NetworkManager behavior looks wrong to me: it should do nothing to IPv6
> > when method is "ignore" and connection is marked as "external": attached
> > patch fixes this behavior.
> > Note that if/once IPv6 address is added, IPv6 method is changed to "manual"
> > and a link local address is generated if needed.
> > 
> > Please review the patch, thanks.
> 
> The description here is insightful. I think it would be good to improve the
> commit message to contain parts of this text. The comment in bugzilla is
> rather detached from the code.
> 
> In general, I agree, we should not touch
> nm_device_sys_iface_state_is_external() devices *at all*. Step by step we
> should fix all the places where we do.
> 
> I find the extra parentheses around the condition unnecessary, would however
> prefer wrapping the line as we do usually, with "&&" at the start.
> 
> 
> Anyway, patch lgtm.

Thanks, updated both the comment and the format of the condition.

Note You need to log in before you can comment on or make changes to this bug.