Bug 1470930

Summary: NetworkManager clears IPv6 PMTU cache when ICMPv6 Router Advertisement is received
Product: Red Hat Enterprise Linux 7 Reporter: Jeremy Visser <jeremyvisser>
Component: NetworkManagerAssignee: Beniamino Galvani <bgalvani>
Status: CLOSED ERRATA QA Contact: Desktop QE <desktop-qa-list>
Severity: low Docs Contact:
Priority: low    
Version: 7.3CC: atragler, bgalvani, fgiudici, lrintel, rkhan, sukulkar, thaller, vbenes
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 13:27:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Reproducer script
none
[PATCH] default-route: skip addition when the route already exists
none
[PATCH v2] default-route: skip addition when the route already exists none

Description Jeremy Visser 2017-07-14 03:26:00 UTC
Description of problem:

If NetworkManager is configured to manage IPv6 (i.e. IPV6INIT=yes) on an interface where Router Advertisements (RA) are used, each time an RA is received, the PMTU cache is flushed.

The impact of flushing the PMTU cache means if the Linux machine is actively communicating with another machine behind an MTU bottleneck, each time an RA is received it will start sending too-large packets, and needs to rely on another ICMPv6 Packet Too Big message before it can send again.

Depending on how frequent RAs are sent on the network or the ICMP rate-limiting behaviour of the intermediate router, the right circumstances can cause several-second hangs in TCP connections.

Version-Release number of selected component (if applicable):

NetworkManager-1.4.0-20.el7_3.x86_64

How reproducible:

This is easily reproducible on EL7 when NetworkManager is set to IPV6INIT=yes and an MTU bottleneck exists on an intermediate network (not the EL machine's directly connected network) and ICMP Router Advertisements are sent frequently.

Steps to Reproduce:

1. Set up EL7 with an IPv6 connection using Router Advertisements (not stateful DHCPv6) and NetworkManager managing the network interface (this is the default configuration).

2. To make the issue easier to reproduce, configure the router next to the EL7 box to sent RAs every 5 seconds.

3. Configure an MTU bottleneck on the far side of the connection, such that when the EL7 box tries to send traffic, it receives an ICMP Packet Too Big message in response.

4. Send some data from the EL7 box to the other box:

    el7$       nc otherhost 9999 < /dev/urandom
    otherhost$ nc -l -p 9999     > /dev/null

5. Perform a packet capture to observe what the resulting behaviour is from receiving an ICMP Packet Too Big, and an ICMP Router Advertisement.

6. Run this command to view the route cache entry for the remote host:

     $ remote_host=2001:db8::1337
     $ while [ 1 ] ; do ip -6 route get $remote_host ; sleep 0.2 ; done

7. Now try again by reconfiguring NetworkManager to not manage IPv6. (i.e. change IPV6INIT=no).

Actual results:

The output of "ip -6 route get $remote_host" will flap between these two values depending on whether an ICMP Packet Too Big or an ICMP Router Advertisement was received most recently:

  good)  2001:db8::1337 via fe80::1  dev ens192  cache  expiry 599  mtu 1398
  bad)   2001:db8::1337 via fe80::1  dev ens192  cache

Expected results:

I expect receiving an ICMP Router Advertisement to be an idempotent operation. Sure, if a route changes, I fully expect the route cache to be cleared.

However, if the RA contains no new information, and the kernel already has the correct table, it should NOT be touched.

Poking the kernel route table when no changes are necessary is considered harmful.

Comment 2 Beniamino Galvani 2017-07-31 19:02:36 UTC
Created attachment 1307217 [details]
Reproducer script

Comment 3 Beniamino Galvani 2017-08-01 07:16:14 UTC
Created attachment 1307398 [details]
[PATCH] default-route: skip addition when the route already exists

Comment 4 Thomas Haller 2017-08-18 15:36:00 UTC
(In reply to Beniamino Galvani from comment #3)
> Created attachment 1307398 [details]
> [PATCH] default-route: skip addition when the route already exists

patch (and approach) locks good to me for nm-1-8 branch.

But there are conflicts with master.

Comment 5 Thomas Haller 2017-08-18 15:37:07 UTC
ok, one thing: 

+                   _LOGt (AF_INET6, "update platform route: %s",
+                          nm_platform_ip6_route_to_string (plat_rt, NULL, 0));
+                   _LOGt (AF_INET6, "                 with: %s",
+                          nm_platform_ip6_route_to_string (&rt, NULL, 0));

maybe not log two lines? :) I know, it's only --with-more-logging, but our logging is already so verbose, please combine the lines.

Comment 6 Beniamino Galvani 2017-08-28 14:23:32 UTC
Created attachment 1319087 [details]
[PATCH v2] default-route: skip addition when the route already exists

Patch rebased on master.

Comment 9 Vladimir Benes 2018-01-29 13:26:25 UTC
Automated long time ago and running since then. Working well in 7.5 too.

Comment 12 errata-xmlrpc 2018-04-10 13:27:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0778