Bug 1461282 - kernel: ICMP rate limiting is too aggressive on loopback
Summary: kernel: ICMP rate limiting is too aggressive on loopback
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Jesper Brouer
QA Contact: Jianlin Shi
URL:
Whiteboard:
Depends On: 1458542
Blocks: 1469549 1469551
TreeView+ depends on / blocked
 
Reported: 2017-06-14 06:32 UTC by Florian Weimer
Modified: 2018-04-10 20:48 UTC (History)
9 users (show)

Fixed In Version: kernel-3.10.0-703.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1458542
Environment:
Last Closed: 2018-04-10 20:47:23 UTC
Target Upstream Version:


Attachments (Terms of Use)
localhost-icmp.c (1.63 KB, text/plain)
2017-06-14 06:32 UTC, Florian Weimer
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1428684 0 medium CLOSED RFE: Backport of ICMP ratelimit fixes. 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHSA-2018:1062 0 None None None 2018-04-10 20:48:38 UTC

Internal Links: 1428684

Description Florian Weimer 2017-06-14 06:32:19 UTC
Created attachment 1287532 [details]
localhost-icmp.c

+++ This bug was initially created as a clone of Bug #1458542 +++

Description of problem:

Aggressive ICMP rate limiting on the loopback interface causes test suite failures.

Version-Release number of selected component (if applicable):

kernel-3.10.0-677.el7.x86_64

How reproducible:

Always.

Steps to Reproduce:
1. Compile localhost-icmp.c.
2. Run ./localhost-icmp

Actual results:

Test runs into a timeout:

localhost-icmp: iteration 50: no ICMP message (poll timeout)

Expected results:

Test completes within a few milliseconds.

Additional info:

https://sourceware.org/ml/libc-alpha/2017-06/msg00167.html
http://marc.info/?l=linux-netdev&m=149656032817085

Comment 1 Jianlin Shi 2017-06-14 08:36:57 UTC
the rate is limited by net.ipv4.icmp_msgs_burst. the default value for it is 50, so you can change its value to 1000 or change ITERATIONS to 50 in your code.

sysctl  -w net.ipv4.icmp_msgs_burst=1000 && ./localhost-icmp

Comment 2 Florian Weimer 2017-06-14 09:33:37 UTC
(In reply to Jianlin Shi from comment #1)
> the rate is limited by net.ipv4.icmp_msgs_burst. the default value for it is
> 50, so you can change its value to 1000 or change ITERATIONS to 50 in your
> code.
> 
> sysctl  -w net.ipv4.icmp_msgs_burst=1000 && ./localhost-icmp

Sure, but the default is way too low, and it is far too easy to hit the rate limit.  In fact, I expect that this will introduce boot delays in some setups (such as those using NIS or other UDP-based RPC services on lcoalhost).

Comment 3 Jesper Brouer 2017-06-14 09:40:36 UTC
Notice I did send a kernel RFC patch, that fix the issue:
 http://lkml.kernel.org/r/20170604163812.602cc089@redhat.com

I'm just not 100% convinced why localhost/loopback traffic require ICMP messages to be excluded from rate limiting?

Sure the reproducer and libc test-case demonstrate that the ratelimiting "works". 
But what use-case need this?

Notice that RFC1812 specify that we _should_ implement rate-limiting, but I cannot see that it recommend to excluded loopback traffic:

 https://tools.ietf.org/html/rfc1812#section-4.3.2.8

Comment 4 Florian Weimer 2017-06-14 09:55:44 UTC
(In reply to Jesper Brouer from comment #3)
> Notice I did send a kernel RFC patch, that fix the issue:
>  http://lkml.kernel.org/r/20170604163812.602cc089@redhat.com
> 
> I'm just not 100% convinced why localhost/loopback traffic require ICMP
> messages to be excluded from rate limiting?
> 
> Sure the reproducer and libc test-case demonstrate that the ratelimiting
> "works". 
> But what use-case need this?

System boot without delays?  The ability to test ICMP-based failover?  (Non-loopback ICMP has been broken for a much longer time; I guess subtleties like these are one reason why people use load balancers with custom IP stacks.)

And isn't this question backwards?  What benefit does rate limiting on localhost bring?  What is the advantage when applications wait for a reply that never comes, instead of learning immediately via ICMP that something is wrong?

> Notice that RFC1812 specify that we _should_ implement rate-limiting, but I
> cannot see that it recommend to excluded loopback traffic:
> 
>  https://tools.ietf.org/html/rfc1812#section-4.3.2.8

That RFC talks about routers.  And in practice, the ICMP handling requirements for routers turned out to be quite different, see RFC 2644.

Comment 5 Jesper Brouer 2017-06-14 11:39:59 UTC
Okay Florian, you convinced me, I posted an upstream patch:

 http://lkml.kernel.org/r/149743965710.19877.13901728209731997446.stgit@firesoul
 http://patchwork.ozlabs.org/patch/775744/

Lets see if upstream people will object to this...

Comment 6 sushil kulkarni 2017-06-16 13:40:49 UTC
After discussions, even though it is a regression, it appears to be a fringe case with no known reported issues. Hence marking it as blocker-. However, recommend fixing this is the next z stream.

-Sushil

Comment 7 sushil kulkarni 2017-06-16 13:42:45 UTC
Just to add..there is also a possible workaround in comment#1.

-Sushil

Comment 9 Rafael Aquini 2017-08-18 22:53:20 UTC
Patch(es) committed on kernel repository and an interim kernel build is undergoing testing

Comment 11 Rafael Aquini 2017-08-21 11:41:56 UTC
Patch(es) available on kernel-3.10.0-703.el7

Comment 13 Jianlin Shi 2017-10-26 00:59:22 UTC
verified on 3.10.0-753:

[root@bkr-hv03-guest30 bz1461282]# uname -a
Linux bkr-hv03-guest30.dsal.lab.eng.bos.redhat.com 3.10.0-753.el7.x86_64 #1 SMP Tue Oct 24 20:20:11 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@bkr-hv03-guest30 bz1461282]# ./localhost-icmp

Comment 17 Jianlin Shi 2017-10-26 10:01:19 UTC
reproducer failed on RHEL-7.4(3.10.0-693):
https://beaker.engineering.redhat.com/jobs/2113308

passed on RHEL-7.5-20171023.n.0(3.10.0-743):
https://beaker.engineering.redhat.com/jobs/2113306

Comment 18 errata-xmlrpc 2018-04-10 20:47:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1062


Note You need to log in before you can comment on or make changes to this bug.