Red Hat Bugzilla – Bug 1461282
kernel: ICMP rate limiting is too aggressive on loopback
Last modified: 2018-04-10 16:48:39 EDT
Created attachment 1287532 [details] localhost-icmp.c +++ This bug was initially created as a clone of Bug #1458542 +++ Description of problem: Aggressive ICMP rate limiting on the loopback interface causes test suite failures. Version-Release number of selected component (if applicable): kernel-3.10.0-677.el7.x86_64 How reproducible: Always. Steps to Reproduce: 1. Compile localhost-icmp.c. 2. Run ./localhost-icmp Actual results: Test runs into a timeout: localhost-icmp: iteration 50: no ICMP message (poll timeout) Expected results: Test completes within a few milliseconds. Additional info: https://sourceware.org/ml/libc-alpha/2017-06/msg00167.html http://marc.info/?l=linux-netdev&m=149656032817085
the rate is limited by net.ipv4.icmp_msgs_burst. the default value for it is 50, so you can change its value to 1000 or change ITERATIONS to 50 in your code. sysctl -w net.ipv4.icmp_msgs_burst=1000 && ./localhost-icmp
(In reply to Jianlin Shi from comment #1) > the rate is limited by net.ipv4.icmp_msgs_burst. the default value for it is > 50, so you can change its value to 1000 or change ITERATIONS to 50 in your > code. > > sysctl -w net.ipv4.icmp_msgs_burst=1000 && ./localhost-icmp Sure, but the default is way too low, and it is far too easy to hit the rate limit. In fact, I expect that this will introduce boot delays in some setups (such as those using NIS or other UDP-based RPC services on lcoalhost).
Notice I did send a kernel RFC patch, that fix the issue: http://lkml.kernel.org/r/20170604163812.602cc089@redhat.com I'm just not 100% convinced why localhost/loopback traffic require ICMP messages to be excluded from rate limiting? Sure the reproducer and libc test-case demonstrate that the ratelimiting "works". But what use-case need this? Notice that RFC1812 specify that we _should_ implement rate-limiting, but I cannot see that it recommend to excluded loopback traffic: https://tools.ietf.org/html/rfc1812#section-4.3.2.8
(In reply to Jesper Brouer from comment #3) > Notice I did send a kernel RFC patch, that fix the issue: > http://lkml.kernel.org/r/20170604163812.602cc089@redhat.com > > I'm just not 100% convinced why localhost/loopback traffic require ICMP > messages to be excluded from rate limiting? > > Sure the reproducer and libc test-case demonstrate that the ratelimiting > "works". > But what use-case need this? System boot without delays? The ability to test ICMP-based failover? (Non-loopback ICMP has been broken for a much longer time; I guess subtleties like these are one reason why people use load balancers with custom IP stacks.) And isn't this question backwards? What benefit does rate limiting on localhost bring? What is the advantage when applications wait for a reply that never comes, instead of learning immediately via ICMP that something is wrong? > Notice that RFC1812 specify that we _should_ implement rate-limiting, but I > cannot see that it recommend to excluded loopback traffic: > > https://tools.ietf.org/html/rfc1812#section-4.3.2.8 That RFC talks about routers. And in practice, the ICMP handling requirements for routers turned out to be quite different, see RFC 2644.
Okay Florian, you convinced me, I posted an upstream patch: http://lkml.kernel.org/r/149743965710.19877.13901728209731997446.stgit@firesoul http://patchwork.ozlabs.org/patch/775744/ Lets see if upstream people will object to this...
After discussions, even though it is a regression, it appears to be a fringe case with no known reported issues. Hence marking it as blocker-. However, recommend fixing this is the next z stream. -Sushil
Just to add..there is also a possible workaround in comment#1. -Sushil
Patch(es) committed on kernel repository and an interim kernel build is undergoing testing
Patch(es) available on kernel-3.10.0-703.el7
verified on 3.10.0-753: [root@bkr-hv03-guest30 bz1461282]# uname -a Linux bkr-hv03-guest30.dsal.lab.eng.bos.redhat.com 3.10.0-753.el7.x86_64 #1 SMP Tue Oct 24 20:20:11 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux [root@bkr-hv03-guest30 bz1461282]# ./localhost-icmp
reproducer failed on RHEL-7.4(3.10.0-693): https://beaker.engineering.redhat.com/jobs/2113308 passed on RHEL-7.5-20171023.n.0(3.10.0-743): https://beaker.engineering.redhat.com/jobs/2113306
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:1062