1624656 – ip -6 route crashes when adding 37 nexthop routes

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1624656 - ip -6 route crashes when adding 37 nexthop routes

Summary: ip -6 route crashes when adding 37 nexthop routes

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	iproute
Sub Component:
Version:	7.5
Hardware:	x86_64
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	---
Assignee:	Phil Sutter
QA Contact:	Jaroslav Aster
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1679911 1679996
TreeView+	depends on / blocked

Reported:	2018-09-02 22:17 UTC by Jonathan Maxwell
Modified:	2019-08-06 12:54 UTC (History)
CC List:	7 users (show)
Fixed In Version:	iproute-4.11.0-16.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	1625358 1679911 1679996 (view as bug list)
Environment:
Last Closed:	2019-08-06 12:54:26 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Article)	3604271	0	None	None	None	2018-09-06 01:04:09 UTC
Red Hat Product Errata	RHEA-2019:2131	0	None	None	None	2019-08-06 12:54:48 UTC

Description Jonathan Maxwell 2018-09-02 22:17:38 UTC

Description of problem:

# ip addr add 1111::200/64 dev br0

# ip -6 route add 3333::/64 nexthop via 1111::1 nexthop via 1111::2  nexthop via 1111::3  nexthop via 1111::4  nexthop via 1111::5 nexthop via 1111::6 nexthop via 1111::7  nexthop via 1111::8  nexthop via 1111::9  nexthop via 1111::10  nexthop via 1111::11 nexthop via 1111::12  nexthop via 1111::13  nexthop via 1111::14  nexthop via 1111::15 nexthop via 1111::16 nexthop via 1111::17  nexthop via 1111::18  nexthop via 1111::19  nexthop via 1111::20  nexthop via 1111::21 nexthop via 1111::22  nexthop via 1111::23  nexthop via 1111::24  nexthop via 1111::25 nexthop via 1111::26 nexthop via 1111::27  nexthop via 1111::28  nexthop via 1111::29  nexthop via 1111::30 nexthop via 1111::31 nexthop via 1111::32 nexthop via 1111::33 nexthop via 1111::34 nexthop via 1111::35 nexthop via 1111::36 nexthop via 1111::37 
addattr_l ERROR: message exceeded bound of 1024
RTNETLINK answers: No such device
*** stack smashing detected ***: ip terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7fd44c1e2677]
/lib64/libc.so.6(+0x118632)[0x7fd44c1e2632]
ip[0x411843]
ip(do_iproute+0x10e)[0x41348e]
ip[0x408384]
ip(main+0x237)[0x407e97]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fd44c0ec3d5]
ip[0x408237]
======= Memory map: ========
00400000-0046a000 r-xp 00000000 fd:01 34003242                           /usr/sbin/ip
00669000-0066a000 r--p 00069000 fd:01 34003242                           /usr/sbin/ip
0066a000-00670000 rw-p 0006a000 fd:01 34003242                           /usr/sbin/ip
00670000-006b8000 rw-p 00000000 00:00 0 
006c6000-006e7000 rw-p 00000000 00:00 0                                  [heap]
7fd44beb4000-7fd44bec9000 r-xp 00000000 fd:01 33554777                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fd44bec9000-7fd44c0c8000 ---p 00015000 fd:01 33554777                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fd44c0c8000-7fd44c0c9000 r--p 00014000 fd:01 33554777                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fd44c0c9000-7fd44c0ca000 rw-p 00015000 fd:01 33554777                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fd44c0ca000-7fd44c28d000 r-xp 00000000 fd:01 33598455                   /usr/lib64/libc-2.17.so
7fd44c28d000-7fd44c48c000 ---p 001c3000 fd:01 33598455                   /usr/lib64/libc-2.17.so
7fd44c48c000-7fd44c490000 r--p 001c2000 fd:01 33598455                   /usr/lib64/libc-2.17.so
7fd44c490000-7fd44c492000 rw-p 001c6000 fd:01 33598455                   /usr/lib64/libc-2.17.so
7fd44c492000-7fd44c497000 rw-p 00000000 00:00 0 
7fd44c497000-7fd44c499000 r-xp 00000000 fd:01 33598461                   /usr/lib64/libdl-2.17.so
7fd44c499000-7fd44c699000 ---p 00002000 fd:01 33598461                   /usr/lib64/libdl-2.17.so
7fd44c699000-7fd44c69a000 r--p 00002000 fd:01 33598461                   /usr/lib64/libdl-2.17.so
7fd44c69a000-7fd44c69b000 rw-p 00003000 fd:01 33598461                   /usr/lib64/libdl-2.17.so
7fd44c69b000-7fd44c6bd000 r-xp 00000000 fd:01 33598448                   /usr/lib64/ld-2.17.so
7fd44c8a3000-7fd44c8a6000 rw-p 00000000 00:00 0 
7fd44c8ba000-7fd44c8bc000 rw-p 00000000 00:00 0 
7fd44c8bc000-7fd44c8bd000 r--p 00021000 fd:01 33598448                   /usr/lib64/ld-2.17.so
7fd44c8bd000-7fd44c8be000 rw-p 00022000 fd:01 33598448                   /usr/lib64/ld-2.17.so
7fd44c8be000-7fd44c8bf000 rw-p 00000000 00:00 0 
7ffdda47f000-7ffdda4a0000 rw-p 00000000 00:00 0                          [stack]
7ffdda5bb000-7ffdda5bd000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

# echo $?
134

Version-Release number of selected component (if applicable):

RHEL7.5
iproute-4.11.0-14.el7.x86_64

How reproducible:

Always as above.

Actual results:

"ip -6 route" crashes.

Expected results:

Should never crash.

Additional info:

Additional routes can be added by multiple "ip -6 route append". But still "ip route" shouldn't crash.

Related to https://bugzilla.redhat.com/show_bug.cgi?id=1624238

in which we are addressing Ipv4 append issue.

Comment 2 Jonathan Maxwell 2018-09-04 00:30:35 UTC

Getting the segment fault varies from machine to machine depending on how many nethop entries are added. The example in the description is on a machine with 4GB RAM. To generate a segfault on a lab machine:

for i in `seq 600`; do
nhs="nexthop via 1111::$i "$nhs
done
ip -6 route add 3333::/64 $nhs
 
rta_addattr_l: Error! max allowed bound 4096 exceeded
rta_addattr_l: Error! max allowed bound 4096 exceeded
rta_addattr_l: Error! max allowed bound 4096 exceeded
rta_addattr_l: Error! max allowed bound 4096 exceeded
rta_addattr_l: Error! max allowed bound 4096 exceeded
rta_addattr_l: Error! max allowed bound 4096 exceeded
Segmentation fault

# echo $?
139

The "ip -6 route" command should have a bounds check for the buffer I think.

Comment 3 Phil Sutter 2018-09-04 17:30:03 UTC

Fix sent upstream: https://marc.info/?l=linux-netdev&m=153608136132035&w=2

Note that this does merely prevent the segfault from happening and establishes a clean error path in that case - it is still not possible to use more than 36 nexthop statements. Please let me know if that is a requirement, I'll then follow-up to have the respective buffers increased.

Comment 4 Phil Sutter 2018-09-04 17:31:39 UTC

Ah, since we didn't update iproute package in RHEL7.6, aiming at RHEL7.7 for this one as well. Z-stream backporting shouldn't be a problem, though.

Comment 7 Phil Sutter 2018-09-06 14:30:29 UTC

I had to respin the patch anyway, but while doing so increased the buffers as well. In a simple case, I could add over 140 nexthops. Though please note that any other parameter reduces the buffer space left for nexthops, e.g. 'src <addr>'. Though since those shouldn't appear in such large numbers, I guess the customer should be able to add up to 128 nexthops in about any case.

https://marc.info/?l=linux-netdev&m=153624072816713&w=2

Comment 8 Jonathan Maxwell 2018-09-06 21:59:00 UTC

(In reply to Phil Sutter from comment #7)
> I had to respin the patch anyway, but while doing so increased the buffers
> as well. In a simple case, I could add over 140 nexthops. Though please note
> that any other parameter reduces the buffer space left for nexthops, e.g.
> 'src <addr>'. Though since those shouldn't appear in such large numbers, I
> guess the customer should be able to add up to 128 nexthops in about any
> case.
> 
> https://marc.info/?l=linux-netdev&m=153624072816713&w=2

That is awesome thanks Phil. If any one wants to add more that ~140 nexthop entries they can use "ip -6 route append".

Comment 9 Phil Sutter 2018-09-16 20:39:21 UTC

jaroslav, please consider this ticket for qa_ack+, thanks!

Comment 17 Phil Sutter 2018-11-16 13:32:18 UTC

Patches to backport:

commit bd59e5b1517b09b6f26d59f38fe6077d953c2396
Author: Phil Sutter <phil>
Date:   Thu Sep 6 15:31:51 2018 +0200

    ip-route: Fix segfault with many nexthops
    
    It was possible to crash ip-route by adding an IPv6 route with 37
    nexthop statements. A simple reproducer is:
    
    | for i in `seq 37`; do
    |       nhs="nexthop via 1111::$i "$nhs
    | done
    | ip -6 route add 3333::/64 $nhs
    
    The related code was broken in multiple ways:
    
    * parse_one_nh() assumed that rta points to 4kB of storage but caller
      provided just 1kB. Fixed by passing 'len' parameter with the correct
      value.
    
    * Error checking of rta_addattr*() calls in parse_one_nh() and called
      functions was completely absent, so with above fix in place output
      flood would occur due to parser looping forever.
    
    While being at it, increase message buffer sizes to 4k. This allows for
    at most 144 nexthops.
    
    Signed-off-by: Phil Sutter <phil>
    Signed-off-by: Stephen Hemminger <stephen>

commit e5da392ff8e3979b86cad04b238ffbbc8076e005
Author: Phil Sutter <phil>
Date:   Thu Oct 18 14:30:31 2018 +0200

    ip-route: Fix for memleak in error path
    
    If call to rta_addattr_l() failed, parse_encap_seg6() would leak memory.
    Fix this by making sure calls to free() are not skipped.
    
    Fixes: bd59e5b1517b0 ("ip-route: Fix segfault with many nexthops")
    Signed-off-by: Phil Sutter <phil>
    Signed-off-by: Stephen Hemminger <stephen>

commit 6cd959bb125c50a04ab6671645fa38c5b07426f4
Author: Phil Sutter <phil>
Date:   Tue Nov 13 16:55:13 2018 +0100

    man: ip-route.8: Document nexthop limit
    
    Add a note to 'nexthop' description stating the maximum number of
    nexthops per command and pointing at 'append' command as a workaround.
    
    Signed-off-by: Phil Sutter <phil>
    Signed-off-by: Stephen Hemminger <stephen>

commit 05d978e0850a6a3bae1e6c5392d82f7b1496f86a
Author: Phil Sutter <phil>
Date:   Tue Nov 13 13:39:04 2018 +0100

    ip-route: Fix nexthop encap parsing
    
    When parsing nexthop parameters, a buffer of 4k bytes is provided. Yet,
    in lwt_parse_encap() and some functions called by it, buffer size was
    assumed to be 1k despite the actual size was provided. This led to
    spurious buffer size errors if the buffer was filled by previous nexthop
    parameters to exceed that 1k boundary.
    
    Fixes: 1e5293056a02c ("lwtunnel: Add encapsulation support to ip route")
    Fixes: 5866bddd9aa9e ("ila: Add support for ILA lwtunnels")
    Fixes: ed67f83806538 ("ila: Support for checksum neutral translation")
    Fixes: 86905c8f057c0 ("ila: support for configuring identifier and hook types")
    Fixes: b15f440e78373 ("lwt: BPF support for LWT")
    Signed-off-by: Phil Sutter <phil>
    Signed-off-by: Stephen Hemminger <stephen>

Comment 26 Jaroslav Aster 2019-03-19 15:11:46 UTC

Hi Phil,

sometimes I see 'netlink allocation error', when I try to add route with a lot of nexthops. Do you have any idea where could be a problem? My test uses dummy interface.

{ ip -6 route add 3004::/64 nexthop via 1111::142 nexthop via 1111::141 nexthop via 1111::140 nexthop via 1111::139 nexthop via 1111::138 nexthop via 1111::137 nexthop via 1111::136 nexthop via 1111::135 nexthop via 1111::134 nexthop via 1111::133 nexthop via 1111::132 nexthop via 1111::131 nexthop via 1111::130 nexthop via 1111::129 nexthop via 1111::128 nexthop via 1111::127 nexthop via 1111::126 nexthop via 1111::125 nexthop via 1111::124 nexthop via 1111::123 nexthop via 1111::122 nexthop via 1111::121 nexthop via 1111::120 nexthop via 1111::119 nexthop via 1111::118 nexthop via 1111::117 nexthop via 1111::116 nexthop via 1111::115 nexthop via 1111::114 nexthop via 1111::113 nexthop via 1111::112 nexthop via 1111::111 nexthop via 1111::110 nexthop via 1111::109 nexthop via 1111::108 nexthop via 1111::107 nexthop via 1111::106 nexthop via 1111::105 nexthop via 1111::104 nexthop via 1111::103 nexthop via 1111::102 nexthop via 1111::101 nexthop via 1111::100 nexthop via 1111::99 nexthop via 1111::98 nexthop via 1111::97 nexthop via 1111::96 nexthop via 1111::95 nexthop via 1111::94 nexthop via 1111::93 nexthop via 1111::92 nexthop via 1111::91 nexthop via 1111::90 nexthop via 1111::89 nexthop via 1111::88 nexthop via 1111::87 nexthop via 1111::86 nexthop via 1111::85 nexthop via 1111::84 nexthop via 1111::83 nexthop via 1111::82 nexthop via 1111::81 nexthop via 1111::80 nexthop via 1111::79 nexthop via 1111::78 nexthop via 1111::77 nexthop via 1111::76 nexthop via 1111::75 nexthop via 1111::74 nexthop via 1111::73 nexthop via 1111::72 nexthop via 1111::71 nexthop via 1111::70 nexthop via 1111::69 nexthop via 1111::68 nexthop via 1111::67 nexthop via 1111::66 nexthop via 1111::65 nexthop via 1111::64 nexthop via 1111::63 nexthop via 1111::62 nexthop via 1111::61 nexthop via 1111::60 nexthop via 1111::59 nexthop via 1111::58 nexthop via 1111::57 nexthop via 1111::56 nexthop via 1111::55 nexthop via 1111::54 nexthop via 1111::53 nexthop via 1111::52 nexthop via 1111::51 nexthop via 1111::50 nexthop via 1111::49 nexthop via 1111::48 nexthop via 1111::47 nexthop via 1111::46 nexthop via 1111::45 nexthop via 1111::44 nexthop via 1111::43 nexthop via 1111::42 nexthop via 1111::41 nexthop via 1111::40 nexthop via 1111::39 nexthop via 1111::38 nexthop via 1111::37 nexthop via 1111::36 nexthop via 1111::35 nexthop via 1111::34 nexthop via 1111::33 nexthop via 1111::32 nexthop via 1111::31 nexthop via 1111::30 nexthop via 1111::29 nexthop via 1111::28 nexthop via 1111::27 nexthop via 1111::26 nexthop via 1111::25 nexthop via 1111::24 nexthop via 1111::23 nexthop via 1111::22 nexthop via 1111::21 nexthop via 1111::20 nexthop via 1111::19 nexthop via 1111::18 nexthop via 1111::17 nexthop via 1111::16 nexthop via 1111::15 nexthop via 1111::14 nexthop via 1111::13 nexthop via 1111::12 nexthop via 1111::11 nexthop via 1111::10 nexthop via 1111::9 nexthop via 1111::8 nexthop via 1111::7 nexthop via 1111::6 nexthop via 1111::5 nexthop via 1111::4 nexthop via 1111::3 nexthop via 1111::2 nexthop via 1111::1 nexthop via 1111::143 encap ip id 42 dst 10.0.0.2; } |& tee /tmp/tmp.GW3v7e64lv'
RTNETLINK answers: Cannot allocate memory

I saw it on rhel-7.5 and 7.6 and all architectures, but not always.

Comment 27 Phil Sutter 2019-03-19 16:16:38 UTC

Hi Jaroslav,

(In reply to Jaroslav Aster from comment #26)
> Hi Phil,
> 
> sometimes I see 'netlink allocation error', when I try to add route with a
> lot of nexthops. Do you have any idea where could be a problem? My test uses
> dummy interface.
> 
> { ip -6 route add 3004::/64 nexthop via 1111::142 nexthop via 1111::141
> nexthop via 1111::140 nexthop via 1111::139 nexthop via 1111::138 nexthop
> via 1111::137 nexthop via 1111::136 nexthop via 1111::135 nexthop via
> 1111::134 nexthop via 1111::133 nexthop via 1111::132 nexthop via 1111::131
> nexthop via 1111::130 nexthop via 1111::129 nexthop via 1111::128 nexthop
> via 1111::127 nexthop via 1111::126 nexthop via 1111::125 nexthop via
> 1111::124 nexthop via 1111::123 nexthop via 1111::122 nexthop via 1111::121
> nexthop via 1111::120 nexthop via 1111::119 nexthop via 1111::118 nexthop
> via 1111::117 nexthop via 1111::116 nexthop via 1111::115 nexthop via
> 1111::114 nexthop via 1111::113 nexthop via 1111::112 nexthop via 1111::111
> nexthop via 1111::110 nexthop via 1111::109 nexthop via 1111::108 nexthop
> via 1111::107 nexthop via 1111::106 nexthop via 1111::105 nexthop via
> 1111::104 nexthop via 1111::103 nexthop via 1111::102 nexthop via 1111::101
> nexthop via 1111::100 nexthop via 1111::99 nexthop via 1111::98 nexthop via
> 1111::97 nexthop via 1111::96 nexthop via 1111::95 nexthop via 1111::94
> nexthop via 1111::93 nexthop via 1111::92 nexthop via 1111::91 nexthop via
> 1111::90 nexthop via 1111::89 nexthop via 1111::88 nexthop via 1111::87
> nexthop via 1111::86 nexthop via 1111::85 nexthop via 1111::84 nexthop via
> 1111::83 nexthop via 1111::82 nexthop via 1111::81 nexthop via 1111::80
> nexthop via 1111::79 nexthop via 1111::78 nexthop via 1111::77 nexthop via
> 1111::76 nexthop via 1111::75 nexthop via 1111::74 nexthop via 1111::73
> nexthop via 1111::72 nexthop via 1111::71 nexthop via 1111::70 nexthop via
> 1111::69 nexthop via 1111::68 nexthop via 1111::67 nexthop via 1111::66
> nexthop via 1111::65 nexthop via 1111::64 nexthop via 1111::63 nexthop via
> 1111::62 nexthop via 1111::61 nexthop via 1111::60 nexthop via 1111::59
> nexthop via 1111::58 nexthop via 1111::57 nexthop via 1111::56 nexthop via
> 1111::55 nexthop via 1111::54 nexthop via 1111::53 nexthop via 1111::52
> nexthop via 1111::51 nexthop via 1111::50 nexthop via 1111::49 nexthop via
> 1111::48 nexthop via 1111::47 nexthop via 1111::46 nexthop via 1111::45
> nexthop via 1111::44 nexthop via 1111::43 nexthop via 1111::42 nexthop via
> 1111::41 nexthop via 1111::40 nexthop via 1111::39 nexthop via 1111::38
> nexthop via 1111::37 nexthop via 1111::36 nexthop via 1111::35 nexthop via
> 1111::34 nexthop via 1111::33 nexthop via 1111::32 nexthop via 1111::31
> nexthop via 1111::30 nexthop via 1111::29 nexthop via 1111::28 nexthop via
> 1111::27 nexthop via 1111::26 nexthop via 1111::25 nexthop via 1111::24
> nexthop via 1111::23 nexthop via 1111::22 nexthop via 1111::21 nexthop via
> 1111::20 nexthop via 1111::19 nexthop via 1111::18 nexthop via 1111::17
> nexthop via 1111::16 nexthop via 1111::15 nexthop via 1111::14 nexthop via
> 1111::13 nexthop via 1111::12 nexthop via 1111::11 nexthop via 1111::10
> nexthop via 1111::9 nexthop via 1111::8 nexthop via 1111::7 nexthop via
> 1111::6 nexthop via 1111::5 nexthop via 1111::4 nexthop via 1111::3 nexthop
> via 1111::2 nexthop via 1111::1 nexthop via 1111::143 encap ip id 42 dst
> 10.0.0.2; } |& tee /tmp/tmp.GW3v7e64lv'
> RTNETLINK answers: Cannot allocate memory

These "RTNETLINK answers:" messages report error status in kernel. I don't know where it comes from, but it seems not directly related to iproute.

> I saw it on rhel-7.5 and 7.6 and all architectures, but not always.

Do you have any statistics how often it happens? Does it only happen with a certain amount of nexthop arguments? If so, what is the minimum nexthop count at which results become unstable?

Thanks, Phil

Comment 28 Jaroslav Aster 2019-03-19 16:39:45 UTC

It happens everytime I run the test, but not always in the same command, but always in ipv6 part of the test. I do not know where is the limit, because I test the maximum possible nexthops for this bug. It happens only in beaker. I'm not able to reproduce it on in 1minutetip.

Comment 32 errata-xmlrpc 2019-08-06 12:54:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2131

Note You need to log in before you can comment on or make changes to this bug.