Red Hat Bugzilla – Bug 668047
/proc/net/ipv6_route broken, causes netstat to segfault
Last modified: 2012-02-21 00:40:56 EST
Created attachment 472278 [details] Proposed patch to fix ipv6_route output Description of problem: The kernel code in net/ipv6/route.c that populates /proc/net/ipv6_route is broken in several ways that result in incorrect output. When netstat tries to parse this output, it segfaults. Specifically: 1. The code that emits the iface name only allows for 8 chars, but iface names may be up to 15 chars (IFNAMSIZ-1). 2. rt6_proc_info() tries to keep the buffer on line boundaries, apparently to compensate for (3) below, but the code is incorrect and results in truncated output. 3. rt6_info_route() overflows the input buffer. This is specifically allowed by proc_read_file() but the overflowing text is truncated by proc_read_file() even when (2) is fixed. Version-Release number of selected component (if applicable): RHEL 5.x (all versions) How reproducible: Always Steps to Reproduce: 1. Create or rename multiple interfaces to use more than 8 chars. 2. Create multiple ipv6 routes such that ipv6_route exceeds 3k. Actual results: 1. netstat crashes: # netstat -nr -A inet6 ... *** stack smashing detected ***: netstat terminated Aborted (core dumped) 2. Lines in /proc/net/ipv6_route have differing lengths. Expected results: 1. netstat does not crash. 2. All lines in /proc/net/ipv6_route have the same length. Proposed patch attached.
Note (1) still exists in RHEL 6.x and in the current git tree. (2) and (3) appear to be fixed as a result of coverting to the seq_file interface.
We probably just need to backport the seq_file interface like rhel6/upstream has done to fix this problem (with the excpetion of the 8 charater name issue), but if this is happening on RHEL6 then that won't help. Can you please post the backtrace of the netstat segfault (or better yet the core file) so that I can see why netstat failed?
I have not actually tried with RHEL 6.x, but after thinking about this a bit more, I believe that converting to seq_file probably fixed all the issues. Because the code in 6.x (and the current git tree) still uses "%8s", the lines will not all be the same length. But the seq_file code doesn't seem to require a fixed line length and neither does netstat, so it should be functional (but a bit deceptive, since the lines all appear to be the same length until you throw it an iface with more than 8 chars). As for 5.x, here is the backtrace from "netstat -nr -A inet6": (gdb) bt #0 0xffffe410 in __kernel_vsyscall () #1 0xf7e30e0a in raise () from /lib/libc.so.6 #2 0xf7e32751 in abort () from /lib/libc.so.6 #3 0xf7e692db in __libc_message () from /lib/libc.so.6 #4 0xf7ef04c1 in __stack_chk_fail () from /lib/libc.so.6 #5 0x08053363 in rprint_fib6 (ext=2, numeric=4) at inet6_gr.c:134 #6 0x080533c1 in INET6_rprint (options=808464432) at inet6_gr.c:257 The function rprint_fib6() is segfaulting because the call to sscanf() near the top of the function is not populating many of the string arguments and this condition is unchecked. That would be inadvisable but functional if the kernel emitted properly formatted lines. My suggested patch makes the kernel emit properly formatted lines. I did not convert to the seq_file interface because of the complexity involved vs. the simple fix -- though I agree, that would probably be the best thing to do.
well, now hold on, I agree that the kernel should probably be converted to use the seq_file interface, but the long and the short of it is that user apps have to be prepared to get back less data then they requested. Its clearly documented by the read function. We can certainly do a seq_file interface from upstream, which will raise the likelyhood that we will read as many bytes as a reader requested, but we still need to modify netstat to cope with the error that you outline above. Re-assigning this to net-tools . I'll open a new bug to do a backport of the ipv6_route seq file conversion.
Certainly netstat should cope with unexpected kernel output. In this case, its only real option is to throw an error message and abort since the kernel is emitting garbage. Interestingly, it looks like someone tried to do some sort of validation on the arguments at some point but it's disabled: #if 0 if (num < 23) continue; #endif But the proper return value would be 31 here, and all 31 arguments need to be filled since the last one is a locally declared string. > the long and the short of it is that user apps have to be prepared to get back less data then they requested. Its clearly documented by the read function. Note that the read function is not used here. The libc functions fgets and sscanf are used, so there should not be any possibility of getting back less data than requested. Either fgets returns a full line or something is very broken. Yes, this is an issue, but IMHO it is very minor and the broken kernel output is a far bigger issue. Please cc: me on the kernel bug also. Thanks!
I wasn't able to reproduce the crash with interface names and route tables that I'm creating. Tom, can you attach a content of /proc/net/ipv6_route that makes netstat crash. Thanks
We have custom hardware and custom compiled packages, so here is a generic repro process using centos (since I don't have an RHEL license handy): * Start with a base install of CentOS 5.5 i386. - Use VirtualBox to install from CentOS-5.5-i386-bin-DVD.iso - Choose text mode install - Choose defaults for install, except: - Select both IPv4 (DHCP) and IPv6 (auto) for eth0 - In "Package selection", uncheck everything for a minimal install - Select reboot when finished - After reboot, immediately exit the setup agent and login as root * Make a script that creates some tunnel interfaces and IPv6 routes #!/bin/sh for n in $(seq 1 15); do x=$(printf "%x" $n) ip tunnel add test-tunnel-$n mode sit local 10.0.0.$n remote 10.1.0.$n dev eth0 ip link set test-tunnel-$n up ip route add 200$x::/64 dev test-tunnel-$n done * Run the script - chmod +x testscript.sh - ./testscript.sh * Note kernel output is fubar: - awk '{ print length; }' /proc/net/ipv6_route * Run netstat and observe abnormal termination with stack smashing message: - netstat -nr -A inet6
Note: tested RHEL 6.0 and verified that the issue does not repro. The kernel output is reasonable (but has unequal line lengths for long iface names) and netstat behaves correctly. Changing the %8s to %15s when generating /proc/net/ipv6_route would make all line lengths equal and give OCD folks warm fuzzies. :)
(In reply to comment #5) > Yes, this is an issue, but IMHO it is very minor and the broken kernel output > is a far bigger issue. Lowering severity then. (In reply to comment #7) > #!/bin/sh > for n in $(seq 1 15); do > x=$(printf "%x" $n) > ip tunnel add test-tunnel-$n mode sit local 10.0.0.$n remote 10.1.0.$n dev eth0 > ip link set test-tunnel-$n up > ip route add 200$x::/64 dev test-tunnel-$n > done This works, thanks.
Created attachment 477027 [details] patch This patch prevents 'netstat -nr -A inet6' from smashing stack.
Great, so it looks like netstat has a fix now. Jiri and Neil, did either of you create a separate bug for the bogus kernel output yet? I'm sure that with IPv6 becoming more relevant it's only a matter of time before more customers start noticing and clamoring for a fix. Thanks!
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Prior to this update, the netstat utility could terminate unexpectedly with a stack smashing message when running the "netstat -nr -A inet6" command and /proc/net/ipv6_route contained lines of different length. With this update, the code that reads the content of /proc/net/ipv6_route is fixed and netstat no longer displays lpExt entries with negative values.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2012-0188.html